How to Build an API-Centric Digital Architecture

This is the first of a two-part series.

Just like with species, the ability to quickly evolve is a key competitive advantage for a digital enterprise — see “Digital Darwinism.” An API-centric approach is a key enabler for agility. In this post, we discuss a typical API-driven architecture, how it can be deployed, and the right tools for the job.

Being agile enables digital enterprises to go to market faster and respond to changes quickly. For example, when developing a product, there is often limited understanding of end user needs. Reducing this risk is not only hard but also expensive; it can involve market research, beta users and expert input. An agile organization can leverage an API-driven approach to obtain a reasonable approximation, learn from users and evolve the system into what users really need.

But, what are some best practices to get it right the first time?

A Sample Architecture 

The following diagram shows how to arrange internal capabilities as APIs and create a digital business by combining and creating new and existing APIs. This is not the only way to do this; however, it is representative of common techniques and ideas.

This architecture breaks the code into two layers: a core API (capability) layer and an experience layer (application layer). The core API layer represents the organization’s capabilities as APIs (aka core APIs).

The experience layer is what end-user applications use directly. These can be websites, mobile apps and APIs exposed to the end user applications and helper services, such as backend for frontend (BFF) services. They reuse core APIs and external SaaS APIs from the API economy.

The experience layer creates value by combining new APIs. APIs in this layer can be consumed by end users or web or mobile apps. This layer can also include event-driven integrations (logic that connects systems, data and APIs together), which are triggered by events happening in the world or events happening in the system.

In a well-designed architecture, the core API layer should evolve slowly and is built by experienced developers employing sophisticated techniques. In contrast, experience layers evolve fast in response to user requests. The goal of this layer is to compose the API from the core API layer and public SaaS APIs and deliver user experiences. This layer requires close collaboration with domain experts and product managers. Accelerating go-to-market and the ability to change while learning from users are critical (growth hacking).

Why does the experience layer evolve faster than the core API layer?

When we need to make a change or add a new experience (UI modifications), some can be handled by reusing APIs without changing them. However, some experience layers will need API changes. A good design aims to keep APIs stable with minimum changes and reduce the need for applications to wait for core API changes. So, the core API layer will change more slowly than the experience layer. The better the API design, the slower the core API layer will change.

Services running in experience and core API layers need several helper technologies. The first is that they need to support user identity and access control, which is supported by customer identity and access management (CIAM) technologies. Second, they need API management (APIM) to expose their services as APIs, handle their subscriptions, throttling and other quality of service (QoS) characteristics and provide support for developers who will use those APIs (typically via a developer portal and sometimes through a marketplace as well).

It is possible to build or assemble both CIAM and APIM using libraries and code, but often the attention, time and money spent building those are wasted, taking you away from your core goals.

It’s not enough to just get the system to production, we need to keep our system running and support it. To do this, we need DevOps and troubleshooting support.

DevOps includes a code repository and a CI/CD pipeline to build and deploy changes and multiple environments (typically dev, staging and production) to test and gradually roll out the code. Some deployments have support for canaries and blue-green deployments to support the gradual rollout and provide an immediate fallback when a problem occurs.

Troubleshooting is a set of tools that helps the support team isolate and fix any problems. At a minimum, this must include observability tools, log management and support to collect data from a running system.

Building an API-Centric Digital Architecture 

We need to decide where we can run the system, how we are going to implement the services and how we are going to expose, manage and govern relevant services as APIs. What you run is a service. Some of those services you may choose to expose as APIs using an API management solution. We can identify two types of services based on their capabilities: business logic services and data services.

Data services are built with wizards or using a programming language, such as Java, Go or Ballerina. Based on the features required, you may choose GraphQL or OData or pure JSON with HTTP. Often, wizards or a low code/no code experience let users explore the database using SQL or other queries and map the results into messages.

Most business logic services would modify a database, retrieve data and run some business logic before responding. They’re often built with languages such as Java (+Spring Boot), Go (go-kit), and Ballerina, which are strongly typed, have solid concurrency models, and are fast.

A notable exception from the above languages is that when the API exposes a machine learning model, users may choose the Python flask framework or TensorFlow Serving.

The services in the core API layer are often written by experienced developers. Those services can run on top of on-premises hardware, in the cloud or on top of a serverless platform. The serverless option can be cost-prohibitive for heavy loads.

Services in the core API layer often need to connect to existing databases, systems or other services in the same layer. They communicate with HTTP, gRPC and sometimes message protocols such as AMQP or Kafka.

Services in the core API layer can be directly used by the experience layer as is or as APIs. If they are exposed as APIs, typically an API management solution (on-premises or provided by a cloud provider) is used.

The experience layer includes APIs, services (such as BFF services or end-user APIs), integrations and also web apps or mobile apps. These applications and services compose other core API layer APIs, services or SaaS APIs. Since the experience layer often focuses on the composition of APIs, it’s mainly built on a connector ecosystem and data mapping capabilities: connectors provide easy-to-use clients to talk to core APIs, services, and SaaS APIs, and data mapping lets us translate between different message formats.

A typical composition API will receive a message directly or through an event, invoke several APIs and translate data between those APIs. The experience layer is often implemented using an Integration Platform as a Service (iPaaS) in the cloud, an on-premise integration tool or using a programming language like Ballerina or Python, or code running serverless. Most of these choices have a connector ecosystem and support data mapping.

Low-code technology used for writing integration (server-side code) is a good choice for the experience layer as it enables rapid development, involvement of domain experts and a wider workforce.

Most services in the experience layer are typically exposed as APIs. If you choose an iPaaS or a cloud platform (such as Azure functions) for deployments, each provides APIM support, observability, log management and sometimes CIAM support. Typically, solutions provided by non-cloud providers tend to be deeper in their features, but you should choose based on your requirements. A few noteworthy APIM solutions include WSO2 API Manager, Apigee or MuleSoft.

Most public APIs, mobile applications or websites provided by the experience layer need to support end-user management, authentication and authorization, which we should solve with a CIAM solution. You can choose between both on-premises or SaaS CIAM solutions such as WSO2 Asgardeo, Okta, and Auth0.

Instead of directly building from scratch or building on top of a cloud provider at the Infrastructure as a Service ( IaaS) level, you can choose to build on top of an internal developer platform like Choreo, which integrates all the above aspects. However, a platform must take an opinionated approach to design decisions and reduce your choices and flexibility, in return for simplicity and more agility.