The Future of Serverless Accelerated by Open Source

Hello everyone, my presentation topic today is "The Future of Serverless Accelerated by Open Source."

Serverless technology is supported by many open-source products and communities, not just cloud provider offerings - this is one of the distinctive features of the Serverless field. Today, I will share how I participate in and contribute to this ecosystem, and what it means for software engineers.

I am Horike, CEO and co-founder of Serverless Operations, Inc. I am also an open-source enthusiast who has created and maintained several open-source projects to help developers in the serverless field. Additionally, I am one of the organizers of Serverless Meetup Tokyo, and before the pandemic, we hosted Serverless Days Tokyo annually.

Serverless Operations, Inc is a Japanese company dedicated to maximizing cloud value through serverless technology. We provide AWS serverless consulting and development services to many companies, primarily in Japan, ranging from large enterprises to small startups.

As you know, almost all Serverless products are pay-per-use. This is beneficial for many companies, especially startups, as cloud computing costs are typically low. However, when your application has high traffic or data volume, cloud computing costs can become expensive.

I would like to share the value of joining the open-source community in the serverless field and the significant impact it can have on a software engineer.

First, let me introduce some open-source projects we maintain.

AWS Step Functions is one of the main AWS serverless services, allowing you to use Lambda functions, ECS, DynamoDB, SQS, Glue, and more to control complex workflows without managing underlying applications or coordinating state.

Here is a real use case from our company: We worked with Lixil's development team to deliver an accounting system. Lixil is a well-known company in Japan with many distribution centers for their logistics business. The project aimed to automatically calculate bills for distribution centers. We recommended using Serverless Step Functions, which achieved automation and significantly reduced costs.

The second open-source tool is the Serverless API Gateway Service Proxy. This is also a Serverless Framework plugin that supports Amazon API Gateway's AWS service proxy integration. You can write an SQS integration like this: if you send a POST request to the API path /sqs, the POST request is directly sent to the SQS queue specified in the QueueName section of the serverless.yml file, eliminating the need to write code just to connect to AWS services. Currently, the plugin supports Kinesis Streams, SQS, S3, SNS, DynamoDB, and EventBridge.

This is our actual usage: WWD JAPAN is a web magazine website for the Japanese fashion industry with a real-time page view ranking feature updated every 10 minutes. Typically, to build such a ranking feature, we use the Google Analytics API. However, it does not support getting rankings from minutes ago, so we decided to use Kinesis Streams as the real-time ranking data store instead.

The architecture works as follows: API Gateway receives visitor view data from the web frontend, data is collected and stored in Kinesis Streams. The Get Ranking API on the right retrieves the collected ranking data from the Kinesis stream and responds to the web frontend. By using the Serverless API Gateway Service Proxy, we established an API connecting API Gateway to Kinesis Streams. Once a visitor visits an article page, the API is called, and the article ID and current time are sent to the Kinesis stream.

The third open-source project is Jeffy, a Python runtime Lambda application framework - a utility suite for Lambda functions that makes developing serverless applications easier. Similar products include DAZN Lambda Powertools and Middy.

Why did I join the open-source community?

That concludes the introduction of our open-source projects. Next, I will explain my background and why I have been working for open source.

I have been a software engineer for about 15 years. About ten years ago, I started using AWS, and AWS Lambda was released in 2014, which had a significant impact on me. Before that, to run code on AWS, you needed to set up infrastructure using EC2 before running the code. But with Lambda, it became possible to just write code - meaning software engineers could simply write code to create new, interesting products on AWS.

In 2015, Amazon API Gateway was released, which was my second major impact. Before that, code could only run on AWS - it was such a closed space. But with API Gateway, new interesting products could be published through APIs and seen by people around the world beyond AWS.

I have always been obsessed with serverless, spending a lot of time playing with API Gateway, Lambda, DynamoDB, and publishing many blog posts with my ideas. When using serverless, I noticed some problems: code on Lambda cannot be uploaded through the AWS Management Console; how can I version my code? How do I create CI/CD pipelines?

At that time, I discovered the Serverless Framework, and I was overjoyed because these problems had already been solved by it. I started communicating with the community and discussing with the core team how it could become a better tool. That's when I started joining the Serverless community.

What is important about participating in open-source communities? Serverless technology is actually supported by many open-source projects, which is a major characteristic of the serverless field.

Why? This doesn't mean cloud vendor products are of poor quality. Serverless technology is not yet mature and will never stop developing. As you know, in most cases, products don't have all the features you need when first released, because cloud vendor products are generally released as minimum viable products. In fact, AWS Lambda in 2014 didn't have some features developers needed - its execution time was limited to a few seconds and couldn't even use important environment features. However, these missing parts gave us great opportunities to fix them through open source.

The Serverless open-source community is so huge. Every day, developers are discussing something to promote product development. In such a huge community, there are many talented developers, and working with them, your development skills will improve. The excellent workflows adopted in open source, such as releases and testing, are also very helpful for your work, so you can learn a lot by participating in the open-source community. Not only that, when participating in Serverless community events, I am happy to meet community members from countries other than Japan.

This is the most important mindset in the open-source community. Imagine your code being used worldwide and bringing happiness to people somewhere. This is a great thing for developers.

Serverless technology is not yet mature, so there are many things that need your open-source ideas to solve. If you are interested in the open-source community, please join us!

Thank you for listening. I am very happy to be here today.