View All Discussion
Referendum#316
Discussion#2612

Subscan | Infrastructure Costs of Subscan for Kusama & Statemine Networks (from January to September 2023)

2 years ago

This is a follow up proposal to what we submitted before:

#229 Subscan | Infrastructure Costs of Subscan for Kusama & Statemine (from July to December 2022)

Subscan is an essential block explorer and analytics tool for the Substrate ecosystem, providing valuable insights to the Substrate community. However, the maintenance and operation of Subscan requires a significant investment of resources, including server infrastructure, security, and personnel costs. Therefore, we propose reimbursing the operation and maintenance fee for Subscan for the period January to September 2023 from the Kusama treasury to ensure its continuous operation and maintenance.

We highly value feedback from our community users, and over the past few months, we've made substantial efforts to reduce infrastructure costs. The results have been both effective and gratifying. Starting from July 2023, the operational expenses have gradually decreased and have recently stabilized. As an illustration, taking our GCP bill as an example, the costs for the past week have decreased by more than 50.52% compared to their historical peak.
For specific parameters and billing details, please refer to the "GCP Bill" section below.
For our specific development efforts, please refer to the "Operations and Maintenance Cost Reduction Plan" section below.

The growth of Subscan is intrinsically tied to the support of our community, which we deeply appreciate. We aspire to enhance community involvement in Subscan's decision-making processes. As a result, we propose a gradual shift in Subscan's development approach, starting from November 2023, towards a community-developer collaborative model. In this new framework, Subscan's development will be guided by the needs and feedback of the community. Features with the highest demand from the community will be prioritized for development, aligning more closely with the ecosystem's requirements. While we have implemented some of these principles in the past, we recognize that they have not been as open and transparent as desired. Our aim is to continually improve upon this aspect. And the specific plan will be officially announced in December!

Please view the detailed proposal here.

Comments (1)

2 years ago

Dear community,

First and foremost, I want to express sincere gratitude for all your valuable feedback. We've received feedback in the comments section regarding the perceived "expensive" nature of our services, and I completely understand the importance of responsibly utilizing the treasury funds for the community's benefit. As the Head of Subscan, I am more than happy to address this matter comprehensively.

We are well aware of the rapid increase in our GCP (Google Cloud Platform) bills and have invested significant efforts to mitigate these costs. Here are some key optimizations we've implemented:

  1. Database Optimization:

    • Optimized block index
    • Enhanced event index
    • Improved extrinsic index
    • Streamlined transfer index

  2. Database Cleanup:

    • Removed redundancy in blocks: extrinsics, logs, and event fields
    • Eliminated redundant log_index in logs
    • Removed redundant fields such as extrinsic block_hash and extrinsic_index
    • Cleared redundant event_index
    • Data cleanup for prod-a and prod-c

  3. Infrastructure Consolidation:

    • Merged prod-c data into prod-a to reduce the number of database instances.

  4. Node Service Testing and Migration:

    • Testing and migration to Onfinality

  5. Resource Optimization:

    • Adjusted CPU and memory usage
    • Replaced some machines for cost-effectiveness

  6. Data Backup Enhancement:

    • Transitioned backup data to more cost-effective storage

  7. Service Log Improvements:

    • Implemented a self-hosted EFK (Elasticsearch, Fluentd, Kibana) log center
    • Transferred most service logs to the self-hosted log center to reduce Datadog costs
    • Discontinued GCP log collection for the cluster, reducing GCP logging and monitoring expenses

As a result, our GCP bill for the past week has decreased by more than 50.52% compared to its historical peak.

Although it may still seem high, please note that this is the total cost for nine months! Subscan, with over 1.34 million unique visitors and more than 495.52 million requests per month, operates within a reasonable cost range. Additionally, this does not cover all our expenses.

Subscan prioritizes data security as any issues with data could lead to misunderstandings about the Polkadot network among users. Considering Polkadot/Kusama's prominence as a blockchain network, investing in high-quality data services is crucial for ensuring data security.

Regarding the suggestion to replace GCP with a more affordable solution, we did consider transitioning to a self-hosted Kubernetes cluster. However, achieving a complete replacement of GCP would require additional operational and development resources, along with substantial testing time. The associated costs were not trivial, and currently, we lack the necessary conditions for such a transition. Given that GCP is among the world's most professional services, and without a better alternative, I believe continuing to use GCP is the optimal solution.

Thank you all for your valuable suggestions. We are committed to continuous optimization, cost reduction, and timely communication of our progress with the community.

PleaseLogin to comment
    Polkassembly - Subscan | Infrastructure Costs of Subscan for Kusama & Statemine Networks (from January to September 2023) #2612