Building api architecture on modern AWS events-based cloud infrastructure

I build with NodeJs and usually using expressjs for the backend and ReactJS with Redux and immutable js for the frontend. I like this pattern especially on UI side, while immutability allows to avoid a lot of bugs, it makes more opinionated environment and allows to focus more on business logic part. During last years polished the UI infrastructure to the point where I can now build tools to generate hardest UI parts like global state managing.

I want same thing for the backend. Ability to build APIs where you can apply time-travel, easily cover with tests, modularize everything to one file = one function level. I also tend to run everything more and more on AWS cloud where API call either REST or Websocket based is just an event. Some JSON which lands to some AWS service, URL endpoint or Lambda. Inside lambda I now run expressjs, which is just… *silence*. There is nothing wrong about expressjs and I used it for many years, never had need to get rid of it but I was not using cloud infrastructure to run things either. Some linux box with nginx and pm2 in AWS EC2 is not real cloud infrastructure, it’s just linux box in the cloud.

Catching the incoming HTTPS/WSS event

So, what’s the best way to run API in cloud native structure, AWS style? Any call to API endpoint hosting service aka AWS ApiGateway is an event. So that event is either piped to AWS service like SQS or SNS where you don’t need live response but you must ensure you can take in the event and respond to user asap. SQS/SNS can scale up to infinity and there is nothing to manage about that. Yet if incoming event is landing to lambda, you manage lambda’s concurrency and what stuff that lambda is calling down the pipe. To make sure API is as stable as possible and responds after doing as less job as possible, first of all incoming event should land to some queue for execution. Probably SQS or SNS. If there is really big traffic at scale to save some $ you may consider to manage Kinesis stream and send those events there. But in general, if you are not Amazon.com SQS/SNS is probably way better option + more managed. Less stuff to think about is always better. Another place to land event is S3 if you have bigger data income, you will have to get S3 upload URL anyways, while API Gateway can consume 10MB and you should not use Lambda to pipe data. It’s computational service not networking. If you have option to send data direct to DynamoDB, that’s popular option too and the last and most flexible option is to forward that event to Lambda.

What happens inside Lambda?

So you received let’s say HTTPS event to lambda from API Gateway. It’s JSON object with all the details, what’s next? You need to spit out another object which will take API Gateway and forward to the original caller.

This part is the interesting one which I’m trying to build now. By lambda nature you get in event and you return event. Aka it’s a function with payload and required response. But inside lambda it’s devs written code.

First line in my lambda is logging full incoming event & context JSON. Last line is just before the return — logging of the result. Target of this is if you have same input, you should be able to get same output. The more you go into this, the more sounds like functional programing.

Functional programing

In functional programming main things are these [longer version]:

1. Functions are pure. So if you call same function many times, for same payload — you get same result back.
2. Functions do not have side effects. Which means any I/O is breaking this. While it’s API and usually you do have I/O with 3rd parties or database it’s tricky to follow this one but on it later.
3. Functions are first-class. This means you can set function to variable and execute it at any place. Sounds like some routing structure?
4. Variables are immutable. Javascript uses pointers all over the place and sometimes it’s easy to change stuff which you don’t want to. On UI side for 5+ years i use immutable js lib from Facebook devs team and it’s great. Will start to use it here too.
5. Referential transparency. It if you can replace the value of a function call with its return value everywhere that it’s called and the state of the program stays the same, then the function is referentially transparent.

Api structure

I like when API is built on these steps:

1. Event handler — pure function which takes in the incoming event and parses it. Easy.
2. Router — pure function which takes in same event and checks which route handler to call. Easy.
3. Middleware — function which is not pure :( while it probably requires database connection, maybe logs event, maybe authenticates this exact user for the action/resource.
4. Handler — function which is not pure :( while it’s core business logic. Can happen anything inside. Mostly CRUD to database, some math. Maybe some 3d party calls.

Middleware & Handlers

Middleware & Handlers are much harder to make as pure as possible while they usually have external I/O to database, files, APIs.

Any change operation should be idempotent, so you could call the api endpoint many times and result is the same. I like the Stripe approach where what they require to support idempotency of the API call is to include Idempotency-Key header and if you call & retry same change operation, it will respond with same response. So on network issue, you can retry safely with knowing it won’t create/update resource once again. Will use same approach, any change operation will require idempotency key.

Database itself could be immutable, especially for money-related data, to have verifiable proof of no data changes because of bug or security issues. Will try to use AWS QLDB (Amazon Quantum Ledger Database) to ensure immutability and track-record of changes to data when designing invoicing service.

Any push-notification/webhook call to 3rd party should happen through SNS while it will handle retries and you don’t need pay for networking time while running lambda.

REST vs Websocket.. GraphQl?

While everything is an event, websocket based APIs seems way better fit to the full structure then REST. During recent years I think more and more about full events based stack from database operation to button click on UI. WS are great for saas, while they provide universal state of the app for all users at once and with services like AWS ApiGateway it is easier than ever to create WebSocket API, so will consider moving all my toolset to socket based architecture and leave REST as an option to integrate with some 3rd party. E.g. to provide webhooks. Yes, i’m aware of graphql approach yet i don’t feel real need for it yet. It is an upgrade for REST to save some networking and possibly CPU in exchange of complexity. With graphql you always have to define what you want to get from API and I tend to optimise for simplicity and less complex stuff from dev standpoint, even if it will require more CPU/SSD/Network resources. Those resources are cheap and super available nowadays anyways, why optimise on those? If app reaches the state when it starts to make sense to optimise, while $ for cloud resources is much higher then developers expense, then sure or if app actually has real lag issue for the end users, but in 95% cases to me sockets and simplicity of just having defined structure sounds like much better optimisation on UX & DX (developer experience).