AppNet: Expressive, Easy-to-build, and High-performance Application Networks

What is AppNet?

AppNet is a framework designed for constructing high-performance application networks for microservices. At its core, AppNet offers a high-level abstraction that facilitates the creation of expressive and performant application networks. Users can define rich, possibly stateful, layer-7 (RPC) processing through match-action rules. AppNet's compiler compiles these specifications and generates high-performance code by optimizing where and how to execute different RPC processing functions.

Details are available in our arxiv paper and talk. (To be added)

Architecture Overview

AppNet consists of three main components - the AppNet program, the control plane and the data plane. TThe AppNet program orchestrates the network functionality among microservices through a sequence of elements, detailing each element with match-action rules that govern RPC content and element state.

The control plane includes a compiler that processes RPC definitions and chain specifications to produce code modules and a controller has global knowledge (acquired via cluster managers such as Kubernetes) of the network topology, service locations, and available AppNet data plane processors. It provisions network processing on available processors.

The data plane is composed of processors—such as sidecars, middleboxes, and RPC libraries—that execute the elemental operations at a low level. Each processor retrieves the compiled RPC processing logic from the control plane and periodically transmits logs, traces, and runtime statistical data back to the controller.

Supported Data Plane Processors

• mRPC
• Istio(Envoy)
  • Sidecar Mode
  • Ambient Mode
• gRPC (via Interceptors)
  • Only gRPC-go is supported

Getting help

Please get in touch with Xiangfeng Zhu (xfzhu@cs.washington.edu).

Installation Guide

Welcome to the AppNet installation guide. This document provides step-by-step instructions on how to set up AppNet and its dependencies.

Note: The following scripts are tested on Ubuntu 20.04. We plan to add support for other platforms soon.

Cloning the Repository

First, clone the AppNet repo:

git clone git@github.com:appnet-org/appnet.git --recursive
cd appnet

Before you start, make sure you have the following installed:

Requirements:

• Kubernetes (v1.28+)
• Istio (v1.22.x)
  • Note: AppNet's support for istio 1.23.x is in progress.
• Go (v1.22.2)
• Rust (v1.71+)
• protoc
• Conda

Install the CLI

appnetctl is a command line program to manage the AppNet control plane. To install the CLI, follow these steps:

1. Create a Conda environment:

conda create -y -n appnet python=3.10
conda activate appnet

2. Run the installation script:

. ./install.sh

3. Verify the CLI and other tools are running correctly:

user@h1:~/appnet$ appnetctl version
Version: v0.1.0
user@h1:~/appnet$ appnetctl verify
Verifying AppNet installation status...
✔ Python installed.
✔ Rust installed.
✔ Kubernetes installed.
✔ protoc installed.
✔ Istio installed.

4. Lastly, install the CRDs into the cluster:

make install

Requirements

Kubernetes

To install a Kubernetes cluster, we recommend using kubeadm. Follow the steps below:

1. Install the Control Plane:

. ./utils/k8s_setup.sh

2. (Optional) Set Up Worker Nodes:

• First, prepare the worker nodes:

. ./utils/k8s_setup_worker.sh

• Then, join the cluster using kubeadm join. Run the following command on the control plane node to get the join command:

kubeadm token create --print-join-command

Note: if you plan to use a multi-node cluster, make sure you can ssh other nodes from the control plane node.

3. Verify Installation:

kubectl version

For additional installation options (e.g., KIND, Minikube), visit this page

(Optional) We highly recommend installing k9s for visualizing your clutser.

Istio

Istio can be installed in either sidecar mode or ambient mode. Choose the one that best fits your requirements (if you want to use both sidecar and ambient mode, please install using ambient script):

• Sidecar Mode

. ./utils/istio_setup_sidecar.sh

• Ambient Mode

. ./utils/istio_setup_ambient.sh

Go

Install Go by running the following command:

wget https://go.dev/dl/go1.22.1.linux-amd64.tar.gz
sudo rm -rf /usr/local/go && sudo tar -C /usr/local -xzf go1.22.1.linux-amd64.tar.gz

echo "export PATH=$PATH:/usr/local/go/bin" >> ~/.bashrc
source ~/.bashrc

Rust

Install Rust by running the following command:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Protoc

Install the Protocol Buffers Compiler and the necessary Go plugins with these commands:

sudo apt -y install protobuf-compiler

Conda

See this page for installation instructions.

For Ubuntu users:

wget https://repo.anaconda.com/miniconda/Miniconda3-py310_23.3.1-0-Linux-x86_64.sh -O Miniconda.sh
bash Miniconda.sh

QuickStart

This guide will walk you through:

• Deploying a simple echo application.
• Running a simple element chain on the frontend to server communication edge.

Echo Application

The Echo application is a simple application developed using Go and gRPC. The client sends messages to the frontend, which then relays the messages to the Echo server. Finally, the server echoes the request back to the frontend. The architecture is as follows:

Deploy

Run the following command to deploy the echo application.

kubectl apply -f config/samples/echo/echo.yaml

Then, verify the deployment:

user@h1:~/appnet$ kubectl get pods
NAME                             READY   STATUS    RESTARTS   AGE
echo-frontend-6f9cf6db74-tjvfc   2/2     Running   0          14m
echo-server-594b4797d-9t6gn      2/2     Running   0          14m
user@h1:~/appnet$ curl http://10.96.88.88?key=hello
You've hit server-986b8c7c6-725kd

Example element chain

We will deploy the following chain to the frontend to server edge.

Run the AppNet controller

First, you need to run the AppNet controller

make run

For this element chain the AppNet configurations is as follows:

apiVersion: api.core.appnet.io/v1
kind: AppNetConfig
metadata:
  name: sample-echo-sidecar # Name of the AppNetConfig
spec:
  processors: # Processors (sidecar/ambient/grpc)
    - sidecar 
  appName: echo # Name of the application
  clientService: frontend # Name of the client service (must be a valid service in the same namespace as the AppNetConfig)
  serverService: server # Name of the server service (must be a valid service in the same namespace as the AppNetConfig)
  method: echo # Name of the RPC method (defined in the proto file)
  appManifestFile: <APPNET_DIR_PATH>/config/samples/echo/echo.yaml # Path to the application manifest file
  clientChain:
    - name: fault # Name of the first element in the client chain
      file: <APPNET_DIR_PATH>/config/samples/echo/fault.appnet # Path to the fault injection element file
    - name: logging # Name of the second element in the client chain
      file: <APPNET_DIR_PATH>/config/samples/echo/logging.appnet # Path to the logging element file
  serverChain:
    - name: firewall # Name of the first element in the server chain
      file: <APPNET_DIR_PATH>/config/samples/echo/firewall.appnet # Path to the firewall element file
  anyChain:
    - name: metrics # Name of the first element in the any(unconstraint) chain
      file: <APPNET_DIR_PATH>/config/samples/echo/metrics.appnet # Path to the metrics element file
  proto: <APPNET_DIR_PATH>/config/samples/echo/echo.proto # Path to the protobuf definition of client service to server service communication

Next, in a seperate terminal, replace <APPNET_DIR_PATH> with your AppNet directory path and apply this yaml file:

# Via Istio sidecar Mode
sed -i 's|<APPNET_DIR_PATH>|'"$(pwd)"'|g' config/samples/echo/sample_echo_sidecar.yaml
kubectl apply -f config/samples/echo/sample_echo_sidecar.yaml

# Via Istio ambient Mode
sed -i 's|<APPNET_DIR_PATH>|'"$(pwd)"'|g' config/samples/echo/sample_echo_ambient.yaml
kubectl apply -f config/samples/echo/sample_echo_ambient.yaml

# Via gRPC Interceptor (proxyless mode)
sed -i 's|<APPNET_DIR_PATH>|'"$(pwd)"'|g' config/samples/echo/sample_echo_grpc.yaml
kubectl apply -f config/samples/echo/sample_echo_grpc.yaml

You should some logs in the controller indicating it is reconciling, which should finish in a few minutes.

Finally, test the installation by running:

user@h1:~/appnet$ curl http://10.96.88.88?key=hello
You've hit server-6646d696cb-mx95h
user@h1:~/appnet$ curl http://10.96.88.88?key=test
Echo server returns an error.

The test request will be blocked by the firewall element.

Clean Up

When you're finish experimenting with the echo application, uninstall and clean it up using the following command:

kubectl delete all,sa,pvc,pv,envoyfilters,appnetconfigs --all
istioctl experimental waypoint delete --all

Next Steps

• Learn more about AppNet specification.

AppNet Tutorials

Overview

AppNet provides high-level abstractions for developers to realize application networks. The abstractions enable easy specification of desired network functionality and enable optimizations that help realize the functionality with low overhead.

In AppNet, network functionality between a pair of microservices is specified as a chain of elements on the RPC request and response paths. Element processing is specified as match-action rules that operate over RPC fields. RPC fields (both metadata and payload) are represented as key-value pairs, which the elements can then easily read/write without worrying about (de)serialization. The state is also represented as key-value pairs.

AppNet’s compiler takes RPC definitions and chain specification to generate code modules. A module may combine the processing of multiple elements (to reduce invocation overhead) and may also re-order element processing based on inter-element dependencies (e.g., RPC fields read/written). The compiler also determines how to partition the state and if parts of shared state can be local (to reduce synchronization overhead). Finally, to further reduce the overhead, the compiler selectively bypasses the sidecar proxy processing.

Next Steps

• Learn more about AppNet chain specification.
• Learn more about AppNet element specification.

Documentation coming

Documentation coming

Developer's Guide

• Controller
• Compiler

Controller

Modifications to AdnconfigSpec

Changes can be made in this file.

make install

Run controller locally

make run

Compiler

Chain Compiler

sed -i 's|<COMPILER_DIR>|'"$(pwd)"'|g' examples/chain/echo.yaml
python compiler/main.py --spec examples/chain/echo.yaml --backend envoy -v --opt_level no

usage: main.py [-h] -s SPEC_PATH [-v] [--pseudo_property] [--pseudo_impl] -b {mrpc,envoy}
               [--mrpc_dir MRPC_DIR] [--dry_run] [--opt_level {no,ignore,weak,strong}]
               [--no_optimize] [--replica REPLICA] [--opt_algorithm OPT_ALGORITHM] [--debug]

options:
  -h, --help            show this help message and exit
  -s SPEC_PATH, --spec_path SPEC_PATH
                        Path to user specification file
  -v, --verbose         If added, request graphs (i.e., element chains) on each edge will be
                        printed on the terminal
  --pseudo_property     If added, use hand-coded properties instead of auto-generated ones
  --pseudo_impl         If added, use hand-coded impl instead of auto-generated ones
  -b {mrpc,envoy}, --backend {mrpc,envoy}
                        Backend name
  --mrpc_dir MRPC_DIR   Path to mrpc repo
  --dry_run             If added, the compilation terminates after optimization (i.e., no
                        backend scriptgen)
  --opt_level {no,ignore,weak,strong}
                        optimization level
  --no_optimize         If added, no optimization will be applied to GraphIR
  --replica REPLICA     #replica for each service
  --opt_algorithm OPT_ALGORITHM
  --debug               Print debug info

The compiler will automatically install elements on all the nodes and

• Generate attach_all.sh and detach_all.sh in graph/gen if the backend is mRPC.
• Generate manifest files if the backend is Envoy. Use kubectl apply -f to run the application

Element Compiler

Follow these steps if you want to interact with the element compiler directly.

The element compiler convert AppNet program to an IR. From IR, we can infer the element property (used by graph compiler). The element compiler also generates backend code for each element.

python compiler/element_compiler_test.py --element examples/elements/echo_elements/fault.appnet --backend envoy --placement client --proto ping.proto --method_name PingEcho

usage: element_compiler_test.py [-h] -e ELEMENT_PATH [-v] -p PLACEMENT -r PROTO -m METHOD_NAME
                                -b BACKEND

options:
  -h, --help            show this help message and exit
  -e ELEMENT_PATH, --element_path ELEMENT_PATH
                        (Element_path',') *
  -v, --verbose         Print Debug info
  -p PLACEMENT, --placement PLACEMENT
                        Placement of the generated code
  -r PROTO, --proto PROTO
                        Filename of the Protobuf definition (e.g., hello.proto)
  -m METHOD_NAME, --method_name METHOD_NAME
                        Method Name (must be defined in proto)
  -b BACKEND, --backend BACKEND
                        Backend Code Target

Frequently Asked Questions

Documentation coming