In the fast-paced world of technology, orchestrating workflows efficiently has never been more crucial! That's where Temporal comes into play – an open-source framework that can handle complex, long-running applications and workflows.

Why Workflow Orchestration Is Crucial for ShareChat

ShareChat operates a multitude of services and jobs, each written in different programming languages. These services are built to handle substantial data volumes and traffic with ease. Jobs can be scheduled to run periodically or executed on-demand, ensuring top-tier performance, reliability, and robust error recovery with automatic retries in case of hiccups.

So, What are ShareChat's desired features for a Workflow Orchestration Engine?

• Scheduling Power: It should seamlessly schedule long-running jobs and offer on-demand execution, pause, and resume capabilities as per specific needs.
• Multi-SDK Support: Compatibility with multiple programming languages such as Go, Java, Python, TypeScript, and more.
• Fortified Security: The ability to enhance communication security, including payload encryption.
• Observability: A user-friendly dashboard for monitoring workflow executions.
• Flexibility in Logic: The capacity to implement diverse business logic easily, making it adaptable across various teams.

Meet Temporal: Your Workflow Orchestration Solution

"Temporal is a developer-first, open-source platform that ensures the successful execution of services and applications." – Temporal

Temporal is an open-source, stateful, and distributed workflow orchestration framework. It's designed to simplify the development of intricate, long-running applications and workflows. Developed by Temporal Technologies, it's widely embraced in cloud-native and microservices-based architectures. Temporal stands out by allowing developers to craft workflows just like regular code, making it exceptionally developer-friendly and efficient.

Intrigued? Let's delve deeper into why Temporal is an exceptional choice for ShareChat.

Comparing Temporal with Other Orchestration Engines

Let’s compare Temporal with popular workflow orchestration engines like Cadence and Airflow.

Temporal vs. Cadence: The Tale of Two Orchestration Engines

Temporal Vs Cadence

Temporal is the fork of the Cadence Project. The fork is fully open source under the same MIT license as Cadence ~ StackOverflow.

Temporal is the offspring of the Cadence Project, residing under the same MIT license. While it inherits Cadence's features, Temporal adds some exciting twists to the mix:

TLS Support: Unlike Cadence, Temporal takes communication security seriously. It employs mutual TLS (Transport Layer Security) for robust encryption and authentication, safeguarding data in transit.

SDK Support: Temporal offers comprehensive SDK support for multiple programming languages. Cadence, in contrast, primarily supports Go and Java, narrowing its developer audience.

Thriving Community: Temporal boasts an active community with abundant blogs, forums, and a lively Slack Channel, making it easier to find assistance and resolve issues.

Payload Metadata: While Cadence stores activity inputs and outputs as binary blobs, Temporal allows metadata attachment. This feature enables advanced functionality, including dynamic serialisation, efficient compression, and encryption.

For ShareChat, where data encryption, compression, and diverse SDK support are crucial, Temporal emerges as the ideal choice.

Temporal vs. Airflow: A Battle of Orchestration Titans

Apache Airflow is a popular open-source platform known for its ability to automate workflows and scheduling tasks. It is widely used to orchestrate workflows, automate data processing, and manage complex data pipelines. While Airflow excels in scenarios such as ETL pipelines and scheduled task execution, it may have limitations when handling complex business logic that demands customised processing for each task or necessitates the creation of event-driven workflows.

Let's compare the Temporal with Airflow:

Multi-SDK Support: Temporal offers extensive multi-SDK support, allowing developers to use various programming languages. Moreover, it enables the creation of workflows in one language while allowing the implementation of activities in a different language. In contrast, Airflow primarily relies on Python for implementing Directed Acyclic Graph (DAG)-based workflows. This difference in language support can be significant when dealing with teams that prefer different programming languages.

Efficient Workflow Deployment: Temporal offers a streamlined deployment process for workflows thanks to its unique architecture. Workflows and activities can be implemented within a single worker, allowing the same worker to process both activity and workflow tasks. Additionally, using activities from one workflow in another adds to its flexibility. This advantage becomes especially clear when managing multiple projects with conflicting dependencies. Temporal simplifies the workflow setup, even in complex scenarios where dependencies might complicate deployment, contrasting with systems like Airflow.

Handle Complex Business Logic: Temporal's code-centric approach makes it well-suited for implementing complex business logic within workflows. In contrast, Airflow may have limitations when handling intricate workflows that involve distinct processing for each activity or task.

ShareChat finds Temporal the ideal choice due to its multi-SDK support and ease of implementing complex business logic.

Bringing Temporal to Life

Let's talk about setting up your Temporal server to unleash its workflow prowess. You have two options: self-hosting for control enthusiasts or cloud-based services for those who prefer convenience.

Self-Hosting your Temporal Server

Here's how you can roll up your sleeves and set up your own Temporal server:

Step 1: Install Dependencies

Before you embark on your Temporal journey, you'll need some tools to smoothen the path:

• Temporal Cassandra Tool: This tool helps set up the Cassandra database schema.
• Helm Charts: Helm charts are essential for deploying Temporal on Kubernetes.

Step 2: Database Setup

Temporal requires a persistent database to store events. You can choose from options like MySQL, PostgreSQL, or Cassandra. ShareChat's choice is Cassandra because it handles millions of workflows and events efficiently.

Now, Let's see how we can configure Cassandra DB for Temporal.

• Initialise Cassandra environment variable, which will be used later.

1
2
3
4
export CASSANDRA_SEEDS=cassandra_host
export CASSANDRA_PORT=cassandra_port
export CASSANDRA_USER=cassandra_user
export CASSANDRA_PASSWORD=cassandra_user_password

• Create keySpace and define its replication factor RF

1
./temporal-cassandra-tool --ep $CASSANDRA_SEEDS create -k $KEYSPACE --rf $RF

Note: Temporal by default uses LOCAL_QUORUM and LOCAL_SERIAL for read and write consistency

• Setup schema

1
2
3
4
5
./temporal-cassandra-tool -ep $CASSANDRA_SEEDS -k $KEYSPACE setup-schema -v 0.0
./temporal-cassandra-tool -ep $CASSANDRA_SEEDS -k $KEYSPACE update-schema -d ./schema/cassandra/temporal/versioned

./temporal-cassandra-tool -ep $CASSANDRA_SEEDS -k $VISIBILITY_KEYSPACE setup-schema -v 0.0
./temporal-cassandra-tool -ep $CASSANDRA_SEEDS -k $VISIBILITY_KEYSPACE update-schema -d ./schema/cassandra/visibility/versioned

Step 3: Deploy using Helm Chart on K8s

This step involves creating a Kubernetes namespace, updating configuration values, and deploying Temporal with Cassandra support. It grants you the power to control your Temporal server.

• Update configuration values in values/values.cassandra.yaml, and run

1
2
3
4
5
6
7
8
9
10
11
12
helm template --namespace temporal-service \
    --replace -f values/values.cassandra.yaml \
    --set server.replicaCount=1 \
    --set cassandra.config.cluster_size=1 \
    --set prometheus.enabled=false \
    --set grafana.enabled=false \
    --set elasticsearch.enabled=false \
    --set server.config.persistence.default.cassandra.hosts=$CASSANDRA_SEEDS \
   --set server.config.persistence.default.cassandra.keyspace=$CASSANDRA_KEYSAPCE \
   --set server.config.persistence.visibility.cassandra.hosts=$CASSANDRA_SEEDS \
   --set server.config.persistence.visibility.cassandra.keyspace=$CASSANDRA_VISIBILITY_KEYSAPCE \
    temporalcass . > temporal-cassandra-helm.yaml

You can find values/values.cassandra.yaml in hem-chart repository

1
kubectl apply -f temporal-cassandra-helm.yaml

• Deploy temporal-cassandra-helm.yaml by running this command

1
kubectl get pods -n temporal-service

• All four services should be up and running now; you can check the services using

Your Temporal Server is Up and Running

Once you've completed the setup, your Temporal server is ready to work its magic. Now, let's create a simple workflow to see it in action.

Implementing the "UploadPostWorkflow" Workflow

With your Temporal server in place, it's time to dive into a sample workflow implementation. While you haven't configured a DNS for your workflow server yet, you can set up port forwarding on the temporal-frontend service for testing.

Create Workflow Client

Your first step is to create a Workflow Client, your gateway to the Temporal service. It allows you to initiate and interact with workflows from your applications.

1
2
3
4
5
6
7
8
9
10
11
func GetWorkflowClient() client.Client {
	workflowClient, err := client.Dial(client.Options{
		HostPort:  "localhost:7233",
		Namespace: "sharechat",
	})
	if err != nil {
		log.Fatalln("Unable to create client", err)
	}

	return workflowClient
}

Implement “Upload Post” Workflow

Temporal workflows are long-running, stateful processes designed to handle complex, multi-step tasks. In the following code snippet, we define an 'Upload Post' workflow, which performs a specific task within a predefined time frame. This workflow is responsible for initiating the creation of a new video post and, upon successful completion, it triggers a notification to the content creator.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
type UploadPostWorkflowRequest struct {
	CreatorName string
	VideoUrl string
}

func UploadPostWorkflow(ctx workflow.Context, workflowRequestData UploadPostWorkflowRequest) (string, error) {
	activityOptions := workflow.ActivityOptions{
		StartToCloseTimeout: 10 * time.Second,
	}
	ctx = workflow.WithActivityOptions(ctx, activityOptions)

	videoUrl := workflowRequestData.VideoUrl
	creatorName := workflowRequestData.CreatorName

	var result string
	err := workflow.ExecuteActivity(ctx, UploadPostActivity, videoUrl).Get(ctx, &result)
	if err != nil {
		return "", err
	}

    var notificationResult string
	err = workflow.ExecuteActivity(ctx, NotificationActivity, creatorName).Get(ctx, &notificationResult)
	if err != nil {
		log.Info("error in notification activity")
	}

	return result, nil
}

Create “Upload Post” and “Notification” Activity

Activities in Temporal are units of work typically executed by worker processes. In this example, we create a "Upload Post" activity for post upload and a notification activity to send notification on successful post upload.

1
2
3
4
5
6
7
8
9
func UploadPostActivity(ctx context.Context, videoUrl string) (string, error) {
	result := fmt.Sprintf("Post has been created %s!!", videoUrl)
	return result, nil
}

func NotificationActivity(ctx context.Context, creatorName string) (string, error) {
	result := fmt.Sprintf("Hello %s, you have created a wonderful post!!", creatorName)
	return result, nil
}

Create Worker and Register Workflow/Activity

To ensure efficient task management, you'll register each workflow and activity function with the worker. This step helps the worker understand their responsibilities for specific tasks.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
func startWorker() {
	workflowClient := GetWorkflowClient()
	defer workflowClient.Close()

	taskQueue := "upload-post-worker"
	workflowWorker := worker.New(workflowClient, taskQueue, worker.Options{})
	workflowWorker.RegisterWorkflow(UploadPostWorkflow)
	workflowWorker.RegisterActivity(UploadPostActivity)
	workflowWorker.RegisterActivity(NotificationActivity)

	err := workflowWorker.Run(worker.InterruptCh())
	if err != nil {
		log.Fatalln("Unable to start worker", err)
	}
}

Running the Workflow and Worker

The final steps involve running your Temporal workflow and worker, but we'll leave that part for you to explore and experiment with.

Applying Temporal Workflows in Real Projects

Temporal workflows are incredibly versatile and can be applied to a range of projects. Here are some examples:

• Schedule Post Workflow: Allow creators to choose when their posts should appear, make changes, or post them immediately.
• Notification Workflow: Plan notifications to pop up at regular times or when specific events occur.
• Video Encoding Pipeline: Create multiple video encodings to deliver high-quality content under bandwidth constraints.
• Audio Matching Pipeline: Match the audio of new video posts with existing ones.

Stay tuned as we delve deeper into each of these use cases in separate blog posts. For more information, you can refer to the Temporal documentation and other valuable resources mentioned in the references.

References and Credits

• Temporal Docs
• Temporal Helm Documentation
• Cadence Documentation
• Airflow Documentation

Credits - Temporal Blogs, StackOverflow

In the world of complex workflows and orchestration, Temporal is a beacon of efficiency, offering the flexibility, security, and developer-friendliness that ShareChat and many other organisations need to stay ahead in this technology-driven era. So, as you embark on your journey with Temporal, remember that orchestrating the intricate dance of data and processes just got a whole lot simpler, thanks to this remarkable open-source framework.

Happy orchestrating! 🚀🌟

Written By - Harshit Gupta

Additional Credits - Peeyush Agrawal, Swapnil Bari

Learn more about our Engineering Team here.