Sharing Code Between Projects: Lessons Learned In The Trenches
Email Newsletter
Weekly tips on front-end & UX.
Trusted by 200,000+ folks.
Click here to kickstart your project for free in a matter of minutes.
Start with a free demo —
Check the frontend report!
Take a break with a 3D game built on Netlify!About a year ago, we came to a crossroad that changed the way we build software today. Like many other teams, we were working on a few things at a time, developing different projects for our web and mobile applications, with shared ingredients in the form of common Node.js code between our back-end repositoriess and microservices, and common React UI components with some slight visual and functional differences between our apps.
As our team grew and code lines multiplied, we began to realize that with every passing day we were writing the same code over and over again. Over time, it became harder to maintain our code base and develop new features with the same speed and efficiency.
Finally, we decided to find a solution that would enable us to share and sync common components of code between our projects. Here is what we learned along our journey, which eventually gave birth to Bit.
Common Code In The Wild
While Git is great for collaborating on a single repository, sharing code between multiple projects can be more challenging than we think.
To get started, we looked into our own code base to learn how many times we duplicated our own integration to our user service. The unbelievable result was no less than 86 instances. After the initial shock, we started thinking that this must also be happening elsewhere.
We asked some friends working in a few different organizations of various sizes to run a simple copy-and-paste detection on their code base, looking for duplicates of code longer than 100 lines. The result blew us away: On average, more than 30% of their code base was duplicated.
Finally, we decided to look deep into the open-source projects on GitHub, checking for both duplications and re-implementations of a simple isString function in the 10,000 most popular JavaScript GitHub projects.
Amazingly, we found this function was implemented in more than 100 different ways and duplicated over 1,000 times in only 10,000 repositories. Later studies claim that over 50% of the code on GitHub is actually duplicated. We realized we were not the only ones facing this issue.
Looking For A Solution
Before building Bit, we looked for a tool that would help us turn the smaller components that our apps are built from into building blocks that could be shared between our projects and synced across our code base. We also wanted to organize them and make them discoverable for our team. Here’s a short summary of what we learned.
A Micro-Package Arsenal With NPM
At first, we considered publishing all of our UI components, utility functions and smaller modules as packages to NPM. This seemed like the obvious solution for modularity for our software’s building blocks. However, we quickly learned that this solution came with huge overhead.
Trying to publish a few files from our project to NPM forced us to split our repository and create new ones just to share this code. When dealing with hundreds of components, this meant having to maintain and make changes across hundreds of repositories.
We would also have to refactor our code base, removing the newly created packages from their original repositories, boilerplating the packages in the new repositories and so on.
Even then, we had now a simple way to organize these packages and make them easily discoverable to our entire team. Another major problem was the coupling between the packages and the owners of their origin repositories, which made it nearly impossible for other people to quickly make updates to the packages while working on their own projects.
This kind of overhead was too much for us to handle. So, we quickly decided to look for a better way to share our code.
Lerna Monorepos
The next option we came up with was to use Lerna in order to refactor our code base into a few multi-package repositories, often referred to as “monorepos”.
The upside of this solution was that it would allow us to maintain and publish all our packages from a single repository. However, this option, too, came with a set of drawbacks, particularly when working with smaller components.
Choosing this option meant we would still have to effectively keep multiple packages with multiple package.json files, multiple build and test environments and a complicated dependency tree to handle between them. Updating these packages must also go through the main repository, still making it hard to modify these package from other projects when working with a few separate monorepos.
For example, take the popular Material-UI React UI library. Even though it uses Lerna to publish five different packages from the same repository, you would still have to install the entire library to use each of its components. Making changes would still have to go through that project as well, and discoverability for these component didn’t improve.
Monorepos can be great for some cases (such as testing or building a project as a whole) and can definitely work for some teams. However, refactoring your entire code base just to share common code between projects while still having to struggle with the issues mentioned above made us drop this option as well.
Shared Libraries
This option was quickly dropped, too. In a lot of way, it resembles using a CD-ROMs instead of an iTunes playlist. First, it made no sense to force an entire library of React components and an entire utility library and so on on each of our projects.
Secondly, every project using it would be tightly coupled to the development of this library, making it impossible to adjust its components for each project. This becomes most painful when sharing common Node.js code between our microservices, which would now be coupled to the library.
Thirdly, discoverability within the library is bound to be poor and would involve a lot of work with its documentation and usage in different edge cases.
Because it makes very little sense to couple and slow down our development, we try to minimize the use of these libraries as much as possible. Even popular JavaScript utility libraries such as Lodash are working hard to make their smaller components independently available via NPM.
Git Submodules
Finally, we turned back time and looked into working with Git submodules.
“You there. You're thinking about using a Git submodule. DON'T. Just don't. It's not worth it, ever.
— Jeremy Kahn (@jeremyckahn) December 16, 2012”
Git enables you to make one repository a subdirectory of another repository, creating a single working tree for the entire project, so that a repository can utilize code from another repository.
As for many other teams, this solution did not last for us. First, submodules only work on the master branch, which causes problems for rapid development. Secondly, submodules increase coupling between projects, which makes it hard to work on cross-repository assignments. Finally, a submodule repository is oblivious to its own nesting and the existence of dependent repositories.
After trying these different solutions, we realized that it shouldn’t be this complicated. There really should be a simpler way to organize, share and develop components of code from different projects. So, we decided to build it, and called it Bit.
Building Bit
Our vision for a solution was simple: turn our components and modules into building blocks that can be easily isolated from any project, organized in the cloud and used in any project.
When building it, we set a few guidelines for what we needed from the project.
- Make it seamless to isolate and share code components from any project, without having to create new repositories or manually configure build and test environments and dependencies for each component.
- Enable two-way development, so that each component could be changed and updated from any project, while changes would be synced across our code base.
- Make it simple to organize and share our components, while making them discoverable for our entire team with useful visual information.
After hard work and extensive research, in 2017 we released the first version of Bit to GitHub.
How It Works
Bit’s workflow is made of three simple steps:
- The first is to simply tell Bit which components of code you would like to share from your project, and it will immediately start tracking them in all of the projects you share them in.
- You can then tag a version for these components so that Bit automatically defines and locks their dependency tree for both file and package dependencies, and creates an isolated environment for each component to build and test in isolation.
- Finally, you can share the components to the cloud (or your own remote server), where they will be organized, will be made discoverable and can be installed with NPM or Yarn like any other package.
You don’t have to create new repositories, split your code base or refactor a single line of code.
Now comes the really cool part. You can also use Bit to import the components into other projects for further development. Because Bit tracks your components between projects, you can simultaneously develop them from different repositories and sync changes across your code base.
This fast and distributed workflow means you won’t be tied by ownership issues, and you can actually develop the shared code and update changes from any of your team’s projects.
Let’s see an example.
Example: Bit With React UI Components
For this example, let’s pick a common use case: syncing React UI components between apps. Although designed to be reusable, achieving such reusability can be challenging.
Let’s take an example React app on GitHub. It contains eight reusable React UI components and one global styling component. As you can see, Bit was added to the repository (see the bit.json and .bitmap files) to track these components — but not a single line of code was changed in the repository. From there, the components were shared to the corresponding scope on Bit’s free web hub.
As you can see, each of the components is now available to any developer to install with NPM or Yarn or to import into their own projects for further development.
All of the components are organized and can be shared with your team and searched for via a search engine. They are presented with visual rendering, build and test results (you can use premade external build and test environments or create your own), and come with auto-parsed docs so that you can make an informed decision on which components to use.
Once it’s changed from a different project, you can update the component’s version in the scope (which works as a remote source of truth) and sync changes between different repositories.
A short tutorial for React is available for the example project.
Conclusion
Sharing code between projects is vital to building software faster, while making your code base simpler to maintain and develop over time. As more of our applications are built using reusable components such as React and Vue UI components, Node.js modules, simple functions, GraphQL APIs and more, turning them into building blocks for different projects becomes more rewarding.
However, the overhead of splitting repositories, refactoring projects, and modifying components from different projects can make it hard to effectively collaborate and share your code. These are the lessons learned from our own journey towards simple and effective code sharing, making it simpler to share, discover and collaborate as a team while building with our common LEGO bricks.
Bit is an open-source project, so feel free to jump in, suggest feedback or ask anything. Just remember that, at the end of the day, sharing code is always about people and about growing a collaborative culture where people play together to build great things.
Further Reading
- The End Of My Gatsby Journey
- SolidStart: A Different Breed Of Meta-Framework
- How To Build A Group Chat App With Vanilla JS, Twilio And Node.js
- Node.js Authentication With Twilio Verify
(rb, ra, al, yk, il, mrn)
