Subscan | Infrastructure Costs of Subscan for Kusama & Statemine Networks (from April to June 2024)
a year ago
Subscan is an essential block explorer and analytics tool for the Kusama network, providing valuable insights to the Kusama community. However, maintaining and operating Subscan requires a significant investment of resources, including server infrastructure, security, and personnel costs. Therefore, we propose reimbursing the operation and maintenance fees for Subscan for the period from April to June 2024 from the Kusama treasury to ensure its continuous operation and maintenance.
View the detailed proposal here:
https://docs.google.com/document/d/1wEkLKR_TVTJ-WK1JttkO17TeHEzwdkXhaW3duMeOWY0/edit?usp=sharing
Comments (19)
helixstreet is proud and delighted to support this proposal.
Summary
I've voted NAY until an explanation is provided as to whether the period that funds are being requested for is from April to June 2024 (13 week, ~3 month period) or from 1 Oct 2023 to 30 June 2024 (9 months), and the requested amount is updated if necessary to rectify any discrepencies.
Questions
1- In the titles of your proposal it says it covers the period April to June 2024 for Kusama & Statemine networks, and in the "Other Costs" section it mentions they cover 13 weeks from April to June 2024, where overall scope is to reimburse infrastructure (resources including server infrastructure, operation and maintenance, security, and personnel costs to ensure continuous operation) for April to June 2024, a 3 month period.
However, in the "GCP Bill" section of the proposal attachment https://docs.google.com/document/d/1wEkLKR_TVTJ-WK1JttkO17TeHEzwdkXhaW3duMeOWY0, it shows Kusama & Statemine networks comprised a mean of 7.33% of total usage of total server infrastructure GCP cost of from 18 Jun 2024 to 16 July 2024 from around that period, and in one of the screenshots it shows 313,171.27 USD (tax included) (1 Oct 2023 - 30 June 2024 in screenshot), which is a 9 month period instead of a 3 month period.
But your are claiming 7.33% of 313,171.27 USD (tax included) (1 Oct 2023 - 30 June 2024 in screenshot) for the 9 month period, instead of claiming 7.33% of only ~$38k + ~$44k + ~$21k = ~$103k USD for only the 3 month (13 weeks) period Apr-June 2024, which equates to 0.0733 * 103k * 13/12 = ~$8179.06 USD (~$8k USD). So appears that you have accidently tried to request funding from Oct 2023 to June 2024 for the proportion of the "GCP Bill" like you did in your Polkadot proposal https://polkadot.polkassembly.io/referenda/1056, but in this proposal it says you're only asking for ~3 months (13 weeks) of reimbursement from April to June 2024.
You are also claiming 7.33% of $27,130.58 for 13 June 2024 - 30 June 2024, shown in the screenshot, which equates to ~$1988.67k USD (~$2k USD).
So the total GCP cost of the server infrastructure for those networks for the ~3 months (13 weeks) should be 7.33% of ~$130k (i.e. ~$103k + $27k) which equals ~$10k USD.
Note that previously you took 7.33% of $340,301.85 USD (tax included) that equated to 24,944.12 USD (tax included), but that is incorrect as it covers ~9 months instead of ~3 months.
So instead of applying for 56,144.12 USD from the Kusama Treasury, I believe only approx. ~$41,200 USD should be requested (i.e. ~$10k USD for 13 weeks of the GCP bill instead of 36 weeks (9 months) of it + 31,200 USD (tax included) for 13 weeks of personnel from the "Other Costs" section).
2- Have you really been sufficiently responsive to issues from the community, handling user feedback from various channels and GitHub on a daily basis? Who handles those requests? For example this issue https://github.com/subscan-explorer/subscan-issue-tracker/issues/44 from Feb 2023 still appears to be open, why is it still outstanding? What about the other ~10 issues that are tagged as bugs https://github.com/subscan-explorer/subscan-issue-tracker/issues
3- Where does it detail in this Kusama proposal the efforts that your personnel have made to reduce infrastructure costs? Is it just related to reducing Datadog expenses and transitioning node services to Onfinality, as mentioned in your other Polkadot proposal? https://polkadot.polkassembly.io/referenda/1056
4- If you've significantly reduced infrastructure costs over past 3 months (13 weeks) has that affected your ability to meet your SLA? What level of uptime have you maintained during that 13 week period from Apr-June 2024? Did you maintain your SLA each month over that period? When did it slip below 99% and if so will you be crediting a percentage discount to future monthly customer bills? https://pro.subscan.io/pricing
@ltfschoen
Thank you very much for your thorough review and the valuable questions you've raised. Your diligent examination will significantly reduce the potential wastage of treasury funds and is of immense importance to the entire community. We truly appreciate having more community members like you who are conscientious and responsible in governance. Witnessing such an engaged community ecosystem is heartening for us.
Expense Issue
Regarding the expense issue you mentioned, it was indeed our oversight. Thank you for pointing it out! We will rectify it as soon as possible.
User Feedback Collection
We primarily gather user feedback through various partner groups and the Subscan Element discussion group. You're welcome to join here: https://matrix.to/#/#subscan:matrix.org.
In these groups, we strive to address and resolve nearly every issue raised. Your feedback and supervision are welcome. This also prompts us to consider whether we should make all issues and our work public, and we will discuss this possibility with the team.
On GitHub, there are indeed some legacy issues that remain unresolved due to reasons such as inability to reproduce, lack of details, or conflicts with the current Subscan design and architecture. At this stage, it is challenging for us to resolve these issues, but we plan to focus more on them in the future, which is why they haven't been closed yet. Thank you for bringing this to our attention; we will prioritize it accordingly.
Infrastructure Cost Reduction
Reducing infrastructure costs involves a comprehensive plan, including but not limited to the following efforts:
Database Storage Structure Optimization:
Database Data Cleanup:
log_indexin logsextrinsic block_hashandextrinsic_indexevent_indexprod-aandprod-cMerging prod-c data into prod-a to reduce the number of database instances.
Node Service Testing and Migration to Onfinality.
Resource Optimization:
c2-standard-16type machines with more cost-effectivet2d-standard-16type machinesData Backup Type Adjustment:
Service Log Adjustment:
Most of this work was completed in the previous cycle, hence not mentioned or billed as additional workload in this cycle. We continue to optimize and aim to further reduce operational costs, with the next plan focusing on alternatives to Cloudflare.
Security and Stability
In our philosophy, security is far more important than cost. This is why we opt for the more expensive GCP over other solutions. We believe that as a crucial gateway in the Polkadot/Kusama ecosystem, security and stability are always our top priorities. I am proud to share Subscan's operational data: https://subscan.statuspage.io.
For users eligible for SLA claims, we proactively offer compensation plans, although no such incidents have occurred to date.