Adrian Zhang is an engineer working on TikTok's frontend monorepo infrastructure. In this post, he shares about a new open source project that was launched this spring by his team.

As TikTok's TypeScript monorepo surpassed 1,000 projects and 200,000 source files this spring, we began to hear complaints about Git slowness. To mitigate this, we tried partial clone, shallow clone, and Git Large File Storage (LFS), but ultimately created a new tool called Sparo. In this post, I'll explain our rationale for developing Sparo, demonstrate how to use it, and highlight its advantages over other sparse checkout approaches.

The problem: Git slowness

How slow was Git? Timings varied significantly depending on network quality, but here's one example.

People with slow internet frequently reported git clone taking more than 40 minutes.

Although techniques like LFS did speed things up, those gains quickly regressed with the addition of new projects every day. It is a problem of scalability: Git stores everything forever, which means a high-traffic repository will steadily increase in every metric: file writes, disk storage, download size. Git will eventually become slow for everyone—it's just a question of when! We hit this problem relatively early due to our rapid iteration and constant experimentation.

Exploring ways to solve Git slowness

To keep pace with repository growth, we combined two advanced Git features:

1. Git sparse checkout: To build a given app, we really only need to check out a project and its dependencies. In a huge codebase, this subset will be relatively small and slow-growing compared to the whole repository. Git sparse checkout provides a way to checkout that subset only and exclude irrelevant files. We further reduce the file set by cloning only your active branch by default.
2. Git partial clone: Even if we reduce the checked out files, Git still fetches all files and their complete revision history since their creation. Git partial clone (--filter=blob:none) optimizes this by fetching file contents on demand and excluding irrelevant history. Unlike shallow clone, the full history is still available if needed. Unlike LFS, partial clone works for any file without requiring a separate storage container.

These are built-in features of Git, so why do we need another tool? When we advised people to configure Git directly, they found it awkward to use. Sparse checkout requires you to determine which folders you need, expressed using cone mode globs that are error-prone. It's feasible to educate a small team about Git best practices, but when you reach 6-digit merge request ID's, things really need to be as simple as possible.

Making a new tool wasn't our first choice... even if it is the most fun choice! We started by looking for an existing solution. The full details won't fit within a single blog post, but a few projects are worth mentioning:

Scalar: Developed by Microsoft, Scalar is effectively a shell command that simplifies Git configuration for scalability. That's because the Scalar team ultimately aspired to improve Git itself. Therefore, most of their work has now been absorbed into official Git. Although we didn't adopt the scalar command line, Sparo would not be possible without Scalar's contributions to Git, which you can read about in Microsoft's blog post.

Josh: The Just One Single History project takes a unique approach by implementing a bidirectional proxy for the Git protocol, so the Git client sees a filtered view of the real repository. This was too disruptive for our established workflows, but it's a clever strategy worth studying.

Focus: Developed by Twitter/X, Focus manages Git sparse checkouts derived from the Bazel build graph. It's used in their internal monorepo that supports over 2,000 engineers, according to a Twitter presentation from BazelCon 2020. Focus was the right basic idea, but our frontend uses PNPM workspaces with RushJS, not Bazel.

A couple enhancements

Sparo follows the spirit of Scalar and Focus, adding a couple other details:

Checkout profiles: For example, if your team normally works on five different projects, the profile can specify to checkout these projects, their dependencies, and maybe some auxiliary folders. Each profile is a JSON file stored in Git. If a stranger needs to fix one of your projects, they can easily discover and checkout your profile.

Mirrored commands: We designed the sparo command-line interface (CLI) to be a drop-in replacement for the git CLI. Intercepting every Git command allows Sparo to ensure that Git is invoked optimally. It also optionally enables collecting anonymous telemetry to power your build team's dashboards. Although a few subcommands like sparo clone and sparo checkout have special profile-aware features, our CLI preserves the familiar interface of Git as much as possible, which greatly facilitated adoption. Sparo is 100% compatible with Git, so you can still use the real git wherever needed.

Sparo: A quick demo

What's it like? Try Sparo for yourself! It's easy:

1. Make sure the latest Git version is installed. Then install our NPM package:

npm install -g sparo

2. For demo purposes, we'll use the Azure SDK project which is a large public RushJS monorepo from GitHub. Instead of git clone, clone with Sparo:

sparo clone https://github.com/Azure/azure-sdk-for-js.git
cd azure-sdk-for-js

The sparo clone command will check out the minimum basic folders and files for the monorepo, which we call the repository skeleton.

3. Create a Sparo profile describing the subset of repository folders for Git sparse checkout:

# Writes a template to common/sparo-profiles/my-team.json
sparo init-profile --profile my-team

Edit the created my-team.json file to select a project to work on:

// common/sparo-profiles/my-team.json
{
  "selections": [
     {
       // This demo profile will check out the "@azure/arm-commerce" project
       // and all of its dependencies:
       "selector": "--to",
       "argument": "@azure/arm-commerce"
     }
  ]
}

The --to project selector instructs Sparo to checkout all dependencies in the workspace that are required to build my-rush-project.

4. Check out the folders and files related to the newly created Sparo profile:

sparo checkout --profile my-team

5. Install the dependencies and start local development. In this demo, only the part of the working tree related to project @azure/arm-commerce was checked out.

rush install

That's it! A more detailed walkthrough can be found in this Getting started guide. If you want to learn more about Sparo's technical optimizations, take a look at Git optimization and sparo auto-config.

Building in the open

Although it seems natural to open source this project, by January, the growing concerns about Git slowness pressured our team to deliver a fix as quickly as possible. We decided to start closed, but pursue open source concurrently as long as those efforts didn't impact our timeline. There were a few weeks where we maintained two forks of our code: one in the internal monorepo, and one under review on its way to GitHub. However, this overhead turned out to be negligible. By the time Sparo 1.0 shipped, we had eliminated the last of the duplicated code.

Working on GitHub brought some important benefits:

• The documentation is more professional and written for a broader audience.
• Engineers seem to write better code when they know it's public!
• We shared our designs and demos with the Rush Stack community, receiving valuable input from senior engineers at other companies.
• TikTok's approval workflow includes review by a security expert, who brought an interesting perspective to this project. We expected that a tool like Git must generally trust its inputs, but Sparo indeed has security expectations.

The road ahead

Besides bug fixes, there are two big features we'd like to build next:

• Telemetry plugin system: I mentioned that Sparo can optionally collect anonymous user activity measurements to power a monitoring dashboard. (This is purely for your own purposes—Sparo does not expose your data to any other company.) Currently, these APIs are experimental and require our users to download a separate package from our private NPM registry. Since each company has different ways of storing such data, we'd like to design a plugin system that makes this easier to implement.
• Support for other frontend workspaces: Since we use RushJS, currently we've only implemented support for Rush workspaces. However, plain PNPM and Yarn workspaces are also fairly popular and would be easy to support as well.

Although Sparo is still in its early stages, the feedback has been overwhelmingly positive. Users generally report that their Git performance concerns are allievated. They say the tool is easy to adopt, which was our biggest concern with this bet. From on-call support, I learned about all sorts of obscure Git features that people use, as well as integrations such as ohmyzsh aliases.

If (or when) your company encounters Git slowness, give Sparo a try!