Treasury proposal: SubBooster, a remote compiling tool for Substrate developers.
Dear community and council:
We are introducing a remote compiling speed up tool which is named SubBooster to Kusama and Polkadot community.
By using the remote compiling service, developers and learners within the eco could compile their projects in a more efficient way that enable them to do it within 10 minutes.
We also integrate some other functions like onchain identity to this service to create more use cases for the network. (more features are listed in the 'Technical Roadmap' section.)
We've made a practical MVP for several developers to test, and we are thrilled to invite more people to test and use it~
Why we make it?
When develop with Substrate, there are a few things that ruin our day :cry:, includes:
- Unacceptable compiling time;
- Weak IDE support;
- No binary smart contract sandbox;
- Complex deployment process, etc.
We have been trying to improve above situations for our own sake, like using remote machine to speed up the compiling. Afterwards, we think it would also be useful for others. So we got the idea to make SubBox which aims to make the development on Substrate more efficient and friendly.
And, As you call tell, SubBooster is the first module of our project.
** How it works**
The current MVP is implemented in following steps,
- user provides name and SSH public key to admin;
- admin creates specific user account and authorizes this key in compiling server;
- user installs cargo remote and configure it to use above compiling server.
- user compiles projects using
cargo remoteand tests the binary by loginning the compiling server.
As you can see, most of the above actions are operated manually. We are going to implement following features in this proposal,
- user registers a web app by using a Kusama/Polkadot address with onchain identity/vote configured;
- user configures the SSH public key and sign a specific message like
Substrate Boxby using the account corresponding above address (polkadot{.js} browser extension needed); - backend service creates user account using the potencial onchain name and authorizes above SSH public key in compiling service;
- further steps is same in above MVP.
The cancellation of register will happen in scenarios,
- user requests cancellation;
- onchain identity is removed;
- user uses the service against the rules.
How to benefit our community?
- It can be a useful tools to help attract more Rust developers to Substrate development.
- It helps to save non-renewable time for Substrate developers and boost their project.
- It will create more use cases for important features such as onchain identify/vote to improve the value of Kusama/Polkadot network
Technical roadmap
Milestone 1: backend service automation and onchain integration.
Period: 2 months.
After the 1st month, early users should be able to submit a form with user and public key information and enjoy the compiling service within 10 mins, without manual operation.
Tasks:
- scripts to deploy and manage the compiling server;
- a web service can save the information provided by the user;
- a scheduled task to register new user in compiling server;
- detail documentation on how to use this compiling service;
- start onboarding users manually to 100 users.
After the 2nd month, users can register the web app with Kusama/Polkadot account and sign a message to prove that they are eligible to use the compiling service.
Tasks:
- Kusama/Polkadot onchain identity integration;
- Integrate polkadot js extension to sign message;
- Backend service verify the message and apply user to compiling server;
- Automatically onboarding 200 users;
Milestone 2: Feature complete and user onboarding
Period: 2 months.
After this milestone, users should be able to apply and cancel free compiling service based on their onchain identity/vote. We also plan to help students who don't have enough ksm to register onchain identity.
Not only Substrate devs, we will also try to reach out Rust community, and provide guidance docs and online communication through Discord, Telegram, Wechat, etc. To ensure more people make best use of this tool.
Tasks:
- benchmark how many users can be supported in single server;
- scripts to scale up compiling servers;
- add promote code logic for students and new comers;
- implement cancellation of service based on the above mentioned rules;
- consulting 3rd party to finalize terms of service.
- Tech support.
Post Milestone 2: Regular server health check and monitor
When we've completed the milestone 1 and 2, we'll keep to provide service for another 2 more months. And after this 2 months, we will try to be keep it alive as a public infrastructure through another proposal with a minimal server cost. (11/18 updated)
Sever type:
After compiling and testing multiple machines, We prepare to use the AMD Ryzen 3950X as the server for its excellent performance on multi-threading capability and good cost performance ratio.
| item | specification |
|---|---|
| Cpu | AMD Ryzen 3950x/16core/32threads |
| memory | 32G DDR RAM |
| Hard Drive | 250GB SSD |
| Customized IP | 1 IP per server |
| Bandwidth | 150TB Bandwidth |
| Port speed | 1 Gbps Port |
server application：
- 5 servers for 1st month
- 10 servers for 2nd - 6th month
Treasury request:
We have 2 developers and 1 part time community manager devote to this project. According the task in the technical map, we've made a [budget in detail.] (https://docs.google.com/spreadsheets/d/1ckMMUDzGKf7QGjv3sKBmHGN6daNktb_KhJgA6Ozf0uU/edit?usp=sharing)
(11/18 updated)
Team introduction
Join for early test
Welcome to join our community.
Please read this guide and apply a test account by post an issue here.
Comments (6)
Hi, can you give us more information about the hardware stack? I am very skeptical in regards to you being able to maintain a good service for a flood of users without significant cost overhead. Keeping 10 minute compilation times will keep a good server at 100% capacity, which means you can serve at most 5-6 clients on a single machine in sequence. This also means you'd have to spin up several machines when there's high demand. I fear this 45k budget would be burned pretty quickly with such an approach.
I like this, but how is it different than just installing cargo remote and cargo on a remote machine? Do you mean you would open source the front end and back end too and your full account management and application stack?
Thanks
Thank you very much for your valuable feedback! @Bruno
Here is my reply about your concern.
The detail is now updated in the proposal in the 'Sever type' section.
Here are the results of a few performance tests:
As we proposed, there will be 10 servers with the same configuration which means the service can support 20 users concurrently.
The concurrent connection session will be limited to 2.
There will be scripts to help user pick server or even automatically connect the idle ones.
We are expecting 200 common users after milestone1, and 500 users after milestone2 along with acceptable user experience. We believe this is pretty practical and achievable in light of not all users are using the servers at the same time.
We propose series of automatic provisioning steps to manage multiple users, onchain identity/vote integration, and proper server configuration. All of them will be open source. Most importantly, we are trying to build a shared economy by leveraging treasury and makes it free to all Substrate users. I'm actually excited to build it.
The backend service will also be open source includes account management and state transition.
The UI/UX will not be open source at first place, but we'll definitely think about it.