dYdX Operations SubDAO 3.0

Operations subDAO Mandate 3.0 Proposal

With a successful second term of the Operations subDAO (“Ops subDAO”) ending in January 2025, the Ops subDAO has created this DRC to request $23.3M of funding from the dYdX Chain community treasury to fund the Ops subDAO operations for an additional 36-month term – Ops subDAO 3.0.

This proposal will highlight the progress of the Ops subDAO in completing the objectives in its 2.0 mandate and explain how the requested funding will be used to support the dYdX Chain community over the next 3 years. The new mandate will allow the Ops subDAO to build on its strengths and expand its responsibilities to provide key infrastructure for the dYdX Chain community.

TL;DR

  • The Ops subDAO is proposing a $23.3m budget for 36 months starting in January 2025
  • Mandate 2.0 deliverables have been completed or are being supported on an ongoing basis
  • Mandate 3.0 will focus on dYdX Chain infrastructure and front ends with technical team expansion to continue improving reliability and resilience
  • The Ops subDAO will first share a text proposal for the dYdX community to confirm the extension and funding of the Ops subDAO mandate and a funding proposal will follow at a later date to implement the approved funding

Context and Team

On June 19, 2023, the dYdX Chain community approved the 2.0 mandate of the Ops subDAO. The 2.0 mandate focuses on ensuring that the dYdX DAO has the resources, tools, and infrastructure it requires to ensure the smooth operations of the dYdX Chain.
So far, the Ops subDAO has achieved the main objectives of the 2.0 mandate with a lean but highly effective team.

5 directors:

  • Web3 Studios GmbH - Areta.io
  • Christoph Sonn
  • Joanna Pope Consulting Ltd.
  • Shaun Musuka
  • Valentin Prossliner (also a core contributor).

1 supervisor:

  • Joseph Axisa on behalf of Axis Advisory

5 core contributors who carry out the day-to-day activities of the Ops subDAO:

  • Valentin Prossliner - Operations Lead

  • Piotr Święs - Site Reliability Engineer

  • Stan Banchev - Community Lead

  • Rudi Mocnik - Technical Associate

  • Ilan Melendez - Technical Project Manager

Second Term Highlights – Ops subDAO 2.0

Key Deliverables

With 4 months remaining of the 2.0 mandate, the Ops subDAO is happy to report that it is currently ahead of schedule in completing the 2.0 mandate. More information about the Ops subDAO’s 2.0 mandate is available below.

While pushing forward these deliverables, the Ops subDAO has improved and continues to improve its internal processes, including but not limited to generating periodic updates and conducting financial audits.

Deliverable Status Details
Indexer Setup Completed / Support Ongoing Engaged a leading service provider, Imperator, to run the indexer since the launch of the dYdX Chain. Imperator continues to go above and beyond to support the dYdX Chain community.
Validator Communication Setup Completed / Support Ongoing Simplified communication setup for dYdX Chain validators, boosting collaboration and information sharing. Provided technical knowledge for chain upgrades by clearly explaining each release, from the proposal stage to voting and the final upgrade block height, ensuring the community is well-informed and aligned.
Web Front End Setup Completed / Support Ongoing Established infrastructure for deploying the open-source dYdX Chain web UI, ensuring new versions are promptly released and maintaining nearly 100% uptime.
iOS Application Setup Completed / Support Ongoing Developed the infrastructure to deploy the open-source dYdX Chain iOS application to the App Store, and ensured timely updates with each new release.
Android Application Setup Completed / Support Ongoing Developed the infrastructure to deploy the open-source dYdX Chain Android application to the Google Play Store, and ensured timely updates with each new release.
User Support Setup Completed / Support Ongoing Engaged user support service provider and onboarded user support infrastructure to provide support to dYdX Chain users on the various front ends.
Operational Setup Completed / Support Ongoing Successfully completed the initial operational setup required for the Ops subDAO to accomplish its mandate, including obtaining credit cards, bank accounts, audit firm and accounting firm. Continuously improve upon the existing setup to satisfy evolving needs and iterative improvements.
Transparency Ongoing Continuously provide periodic updates about Ops subDAO activities on the forums and blogs, including financial updates. Engaged an auditor to conduct a financial audit of the Ops subDAO, slated to be published in Q4 2024.
Legal and Regulatory Strategy Ongoing Continuously improving Ops subDAO’s legal and regulatory strategy through seeking advice from well-established law firms in relevant jurisdictions to best position the Ops subDAO.
SubDAO Playbook Completed / Ongoing Recently published an updated version of the subDAO Playbook, primarily focused on learnings from operating the Ops subDAO.

Finances

The Ops subDAO is targeted to complete the 2.0 mandate within its original budget. As of Sept 1st, 2024, the Ops subDAO has $2.1M USDC, 1.1M DYDX tokens, and approximately 2 ETH remaining.

After the expiration of the 2.0 term on 18th January 2025, we are proposing to roll over any remaining budget to the Ops subDAO 3.0 mandate.

Below is a breakdown of the original budget, the current balances, and a forecast of the budget until the end of the second term.

The differences in the estimated and actual amounts for each budget item are due to the following:

  • With higher-than-expected usage of the indexer (which is a positive development showing the growth of the dYdX Chain), the estimated cost of AWS was higher than expected, accounting for over 50% of the monthly expenses.
  • Additional third-party integrations and vendors, such as Elliptic (AML and sanction screening of wallets), were required for the smooth operation of the front ends and indexer.
  • More personnel were required to ensure the level of high-quality services mandated in the 2.0 term.
  • Additional legal entities, including the DOS and DAS, were required to facilitate Ops subDAO operations and the deployment of certain front ends.

For the remainder of the term, the Ops subDAO forecasts average monthly expenses of roughly $412k (totaling $1.9m for 4 months and 19 days), which includes the short-term planned hires and a projected increase in indexer usage and associated costs. At the end of the second mandate, the Ops subDAO will have, according to our forecast, the following estimated balances in its account:

  1. 108,000.00 USDC
  2. 900,000.00 DYDX tokens

Ops subDAO 3.0 - Focus and Building on Our Strengths

We propose extending the Ops subDAO’s mandate for a third term. During this term, the Ops subDAO will continue to provide the essential resources, tools, and infrastructure needed to ensure the smooth operation of the dYdX Chain. Additionally, the Ops subDAO will continue to optimize the operational setup to support the dYdX Chain infrastructure optimally and reliably.

Operating and Supporting dYdX Chain Infrastructure

The Ops subDAO will continue to support critical dYdX Chain infrastructure for the dYdX community:

  1. dYdX Chain Indexer

The indexer is a vital component of the dYdX Chain that is necessary for the functioning of the various dYdX Chain front ends (web and mobile) and for API traders. This service stores on-chain data from the dYdX Chain and off-chain data from the order book to facilitate database queries so that the necessary information can be easily displayed and used.

The Ops subDAO has engaged a high-quality service provider, Imperator, to deploy and maintain the indexer. Imperator has proven to be highly reliable and able to react quickly to unforeseen events, becoming a crucial partner in solving unforeseen problems.

Under the new mandate, the Ops subDAO will explore ways to improve upon and to optimize the dYdX Chain Indexer, primarily trying to mitigate the cost drivers of the infrastructure the indexer is running on.

  1. Front Ends - dydx.trade, iOS App, and Android App

The different front ends are permanently updated and tweaked to improve the trading experience for users, as Ops subDAO we are striving to increase the rollout speed of these changes to our deployments of these front ends and commit to rigorous testing of our deployments to ensure a smooth trading experience.

Additionally, we plan to implement a seven-day per week on-call service engineer ensuring all systems are up-to-date and operational. Additionally, the Ops subDAO will expand system monitoring and regular testing of all front-end deployments and functionalities.

  1. Validator Communications and Alert Systems

Validators are at the core of the dYdX Chain. Validator communications channels are essential for alerting validators of chain upgrades and other events, and for enabling validators to discuss technical topics, fostering collaboration and improving validators’ average performance. Examples of such systems that are managed by the ops subDAO are the validator Slack channels, emergency alerting system, and the status.dydx.trade webpage.

Target Team Structure & Key Activities

As mentioned above we are five core contributors, three technical all-rounders and two operators. While we are currently looking to hire a QA to add to the technical team and a Legal Counsel. In the new mandate we want to bring on two additional technical contributors adding to the dYdX Chain Unit and the Front-end Unit capacities.

The Ops subDAO’s team is divided into three core units:

  1. dYdX Chain Unit (1 new hire)

The dYdX Chain unit will facilitate validator communications and monitor validators and other dYdX Chain infrastructure, including the indexer, alerting the relevant participants when necessary. The two technical project managers are responsible for supporting the validators with their protocol knowledge, guiding them and other community collaborators through chain upgrades and scheduled maintenance, and assisting with any incident response. An infrastructure specialist will explore improvements to the infrastructure setup of the Ops subDAO and provide expert support where needed.

  1. Front-End Unit (1 new hire)

The front-end unit will deploy the different front ends (dydx.trade, iOS application, and Android application), keeping them up to date, and monitoring their uptime and performance. Two site reliability engineers and a quality assurance engineer will ensure all front ends deployed by the Ops subDAO are fully up-to-date, available, and well-functioning.

  1. Operations Unit

The operations unit will run all of the day-to-day Ops subDAO operations and conduct future strategic planning. This unit includes a legal counsel, operations lead, and operations manager, who will be responsible for all non-technical workstreams – from legal and finance to people and communications. The objective is to continuously improve existing processes and efficiency of the Ops subDAO while engaging in long-term strategic planning with respect to legal and regulatory strategy.

Term Duration

To hire top-tier talent for the Ops subDAO (including the proposed development team), a 36-month third term is necessary for offering competitive compensation and supporting long-term retention. A longer planning horizon will also give the team a strategic advantage and enable more cost-effective operations, especially toward reducing indexer-related costs.

Budget

The breakdown of the $23.3m budget for the next 3 years requested from the dYdX Chain community treasury is as follows:

As detailed below, the majority of the costs are indexer (37%) and personnel (27%) costs with the remainder split among the rest of the budget items.

Indexer (37%)

The indexer is a key component of the infrastructure for the dYdX Chain. The cost of running the indexer fluctuates based on the usage of the dYdX Chain. Because the indexer conducts real-time data ingestion and makes the data readily available to traders, it requires a significant amount of AWS traffic, which is the main cost driver of the indexer. 75% of the cost of the indexer is due to AWS costs, while the remaining 25% covers the indexer service provider that deploys the indexers, and provides 24/7 coverage and support.

We based the average cost of the indexer on the current maximum spent in a month (April 2024) to account for more usage of the dYdX Chain in the next 3 years. With the help of Imperator, we are continuously working on optimizing the cost of AWS.

Personnel (27%)

To accomplish the proposed third mandate, the Ops subDAO needs to attract top-tier talent and grow our team of core contributors. It is paramount to build strong in-house competency and capacity for operating the various technical components, reducing the dependency of the Ops subDAO on third-party vendors and be more cost-effective in the long term.

The budget includes all expenses such as service providers, HR admin services and token allocations.

IT Infrastructure (6%)

This budget item includes all IT infrastructure required for the front ends. To ensure a smooth user experience, this IT infrastructure includes various third-party integrations and vendors to monitor the performance and uptime of all dYdX Chain infrastructure and to collect insights for iterative improvements.

Professional Services (13%)

The Ops subDAO works with a variety of service providers, including legal, tax and accounting advisors, to ensure that the Ops subDAO effectively operates in the most optimized manner in accordance with requirements under applicable law. This budget item also includes a buffer for any unanticipated legal expenses.

Administration (8%)

This budget item comprises the cost for the baseline administration of the different Ops subDAO entities (including the DOF, DOS and DAS), such as compensation for directors and supervisor, regular legal fees, and IT infrastructure for internal operations. Each entity requires its own set of accounting, IT infrastructure, staff, and other administrative costs to ensure the separation of each entity.

Operating Expenses (8%)

This budget item includes transaction fees, bank account fees and other similar operating expenses, and is based on the budget in the prior mandate. A buffer is included to account for any market downturn and unforeseen expenses arising from the additional responsibilities in the third mandate, which had been useful in the prior mandates.

Surplus from Second Mandate

We propose to rollover the remaining budget (approximately 900k DYDX tokens and 64k USDC) from the second mandate as an additional buffer for Ops subDAO 3.0, increasing budget security, especially in light of recent market volatility.

Asset Management Strategy

The Ops subDAO will aim to maintain a healthy runway in stablecoins and stablecoin equivalents to balance against market fluctuations while also exploring different conservative yield strategies with both, stablecoins and DYDX tokens to maximize the effect of the budget granted by the dYdX Chain community treasury.

Proposal Timeline and Next Steps

  1. Publish RFC on the forum on September 6, 2024
  2. AMA on September 13, 2024
  3. Incorporate community feedback
  4. Publish text proposal on the dYdX Chain on September 20, 2024
  5. If the text proposal is successful, proceed with a funding proposal at a later date to implement the approved funding amount in the text proposal

Given the extended time horizon of this particular proposal, the current market situation, and our strong balance sheet, we wanted to separate the commitment of the community to the extension of the dYdX Ops subDAO mandate from the actual funding proposal.

Disclaimer

The content of this document (this “Document”) is provided for general informational purposes only, and does not constitute, and should not be considered, or relied upon as, financial advice, legal advice, tax advice, investment advice or advice of any other nature. You agree that you are responsible for conducting independent research, performing due diligence and engaging a professional advisor prior to taking any financial, tax, legal or investment action related to the foregoing content. The content of this Document is not an offer, solicitation or call to action to make any investment of, or purchase, any financial or crypto instruments, assets or related services, of any kind. The Ops subDAO makes no representation, assurance or guarantee as to the accuracy, completeness, timeliness, suitability or validity of any information in this Document or any third-party websites that may be linked herein.

5 Likes

I think it’s better to wait for the proposal on the treasury subdao to understand if we have anything left in the Community treasury at all. $20M for Chaos, $23M here. Without an influx of new money, the protocol will soon become insolvent.

1 Like

This proposal presents a crucial opportunity to not only secure the necessary funding for the continued operations of the Ops subDAO but also to introduce performance-based compensation mechanisms. By aligning compensation with the tangible results that the protocol achieves, we can better align the interests of all stakeholders within the organization. As outlined in the forum post, the Ops subDAO has done a remarkable job in fulfilling its objectives under the 2.0 mandate, and with a lean but highly effective team, the results speak for themselves. However, as we look toward the 3.0 mandate, I believe it’s important to enhance accountability and motivation by tying rewards to specific milestones and performance metrics.

The proposed budget of $23.3 million over 36 months seems entirely reasonable, especially considering that dYdX is a leader in the derivatives segment. In fact, I would suggest introducing additional incentives that reward the team for achieving key milestones. This practice not only ensures that resources are used efficiently but also allows the protocol to maintain flexibility by rationalizing expenses if the anticipated results are not met. For example, by tying part of the budget to specific growth objectives or usage metrics, we can ensure that the funds are being deployed in the most impactful areas.

Moreover, I agree with the strategy of rolling over the remaining budget from the 2.0 mandate to provide additional financial security, especially in the face of market volatility. The commitment to ongoing transparency and periodic updates is commendable and will be vital in ensuring the community remains informed about how the budget is being utilized and the progress of the Ops subDAO.

At this stage, I believe it’s critical for the project to prioritize financial resources toward initiatives that can generate the most growth, such as expanding the technical team and optimizing the infrastructure, without compromising on the operational aspects that are fundamental pillars of the protocol. A balanced approach that fosters both growth and operational excellence will set dYdX up for continued success over the long term.

In conclusion, the Ops subDAO’s 3.0 proposal is an excellent opportunity to not only secure the necessary funding but also implement best practices in aligning team performance with the protocol’s success. This ensures that we are not only leading in terms of infrastructure and operational setup but also in the efficient use of resources, incentivizing the team to reach key milestones and ensuring the long-term viability of the protocol

1 Like

At Govmos, we offer our full support to the Operation SubDAO and are hopeful that this third proposal will pass the on-chain approval vote.

However, we would like to provide some constructive feedback by pointing out a few parameters that caught our attention, and we would appreciate clarifications before the proposal goes to vote.

First, compared to the 2.0 version, which was proposed to last for 18 months, the current proposal doubles the duration. While we would have expected a similar timeframe renewal, we’re curious to understand the rationale behind renewing the mandate for such an extended period. Secondly, the funding request in the previous proposal was $6.6M for 18 months. This new proposal asks for $23.3M over 36 months, which reflects an annual budget increase of 76.5%. This seems significant, and we haven’t encountered any particular explanation for this increase. Additional details on these two points would help provide clarity.

Lastly, we identified a potential contradiction that may benefit from further clarification for the community to fully understand the rationale behind the projections.

The breakdown of the proposed $23.3M budget includes an expected 7% increase in expenses for the Indexer in 2026, followed by another 11% increase in 2027. This appears to contradict the goal of cost mitigation. We would appreciate further elaboration on the forecasts and the envisioned cost mitigation strategies.


To conclude, we reiterate our full support for the mandate renewal, as we have been impressed with the achievements of the Ops SubDAO so far. We simply believe that addressing these points before the on-chain proposal would benefit the community. We look forward to the team’s feedback on these matters.


Thank you for reading,
Govmos.
pro-delegators-sign

1 Like

Thank you @Govmos for the feedback and the questions, please find my answers below:

As pointed out in the original post under Term Duration: the longer term is providing us with a long enough time horizon to (1) attracting and retaining top tier talent within the subDAO and (2) enable us to plan long term and operate more cost effectively (i.e. long term contractual commitments to service providers for better pricing, upfront payments to secure discounts, etc).

Please excuse if this was not laid out in more detail, the indexer infrastructure cost has more than doubled from the first months of operation until May. This was due to increase in traffic as well as more performant infrastructure to improve service quality/speed. As laid out at the time, through long term commitments and prepayments we were able to stop the increase from continuing in the short term and actually reduce it (The monthly infra spend of August 24 is down almost 30% compared to May 24).
We nevertheless see a continuing trend upward in the infra cost, which is expected as it is related to the usage of the dYdX Chain. We have budgeted for the 3.0 Mandate with the expectation of a substantial increase in the activity on the dYdX Chain while factoring in our efforts to reduce the impact on the cost with our optimization efforts. In other words, without cost optimizing efforts the cost increase for the indexer would be substantially more.

Next to the indexer, which is the largest cost driver, the increased budget is also attributed to the larger team, as laid out necessary to deliver on the increased responsibilities in 3.0, most notably, QA, more detailed monitoring, and on call support for the community owned infrastructure.

Please let me know if this answers the questions, happy to provide more info if needed.

We can ideate on indexer costs which may be reduced by developing api server infra based on full node streaming and order updates via full nodes. Costly indexer may be replaced partially or completely. Long term commitment to costly solutions is not right decision.

Thank you for providing additional clarity on these aspects. From our perspective, the 76% increase still raises concerns. While we fully agree with your stance on addressing the expected rise in indexer costs, we believe the proposed significant increase in workforce expenses may be overstated. We recommend revisiting this assumption and adjusting expectations in that regard.

The proposed 27% budget allocation for personnel expenses is a point of contention for us. We suggest reducing this closer to the 20-25% range, which aligns more with reasonable operational standards. We are, of course, open to understanding the rationale behind maintaining this allocation and look forward to hearing your perspective.

1 Like

I understand your concern @govmos and agree that the increase is substantial.

As for the personnel cost, we are planning to grow our team to support the community owned infrastructure and continuously optimize the infrastructure setup. In practice this enables us to be on call during more hours of the day to limit downtime as much as possible, testing each new deployment rigorously to find possible quirks or bugs as quickly as possible, and build redundancies in our team to eliminate key man risk.

To summarize the increased percentage of spending on personnel is a reflection of the responsibilities the Ops subDAO is proposing to take on in this Mandate 3.0. Deliverables which can’t be delivered on without the necessary top tier talent.

Unfortunately we regret to tell you that we will cast a negative vote unless this critical point is addressed and the proposed personnel budget is readjusted back into a more reasonable bracket. If these changes are performed accordingly, we will be happy to reverse course to a positive vote.

2 Likes

Hey all,

besides working with Imperator on reducing the AWS bill, has any effort been made to explore bare-metal alternatives? Its clear the service provider is not the most expensive part here so no need to go against their backs, but maybe it makes sense to see if there are alternative providers who can create/manage bare-metal alternatives alongside Imperator to one-day switch over?

Running into +2m USD yearly stipends to simple be able to trade and operate the app seems simply unreasonable for a project as permissionless as dYdX.

besides that, thanks for all you do!

Are the 5 core contributors al full-time?

best,
Ertemann
Lavender.Five Nodes

1 Like

Hello everyone,

First, we’d like to acknowledge that as Imperator, we are a partner to the dYdX Operations subDAO and receive funding for our services, specifically in running the indexer.

Working with the subDAO has been a highly professional and collaborative experience, allowing us to contribute to the ecosystem’s resilience and reliability. The impact of the Operations subDAO on the dYdX ecosystem is significant, and their role extends far beyond infrastructure maintenance. They consistently optimize processes, address complex technical challenges, and enhance the overall user experience, directly contributing to the growth and stability of the dYdX Chain.

The subDAO team’s ability to deliver on their commitments ahead of schedule and within budget reflects their dedication to efficient and effective operations. They have successfully managed a complex setup involving critical components like validator communication, user support, and front-end maintenance—all of which require significant expertise and coordination. Their ongoing efforts in transparency, regular updates, and financial accountability further underscore their commitment to the community.

Regarding the indexer architecture, it’s important to clarify that it is specifically designed by the dYdX Trading team to meet the complex needs of the dYdX Chain, including real-time data ingestion and high availability across various front ends. This architecture cannot simply be transitioned to bare-metal setups without significant risks and re-engineering. The current design is optimized for the AWS environment, balancing scalability, performance, and cost-effectiveness as much as possible under the existing infrastructure constraints. For those interested in understanding these complexities, I highly recommend reading the detailed overview provided by the dYdX team here: V4 Deep Dive: Indexer

We are continually exploring cost-saving measures, including working closely with the subDAO and Trading team to find optimizations wherever feasible. Their commitment to enhancing the infrastructure, optimizing resource usage, and engaging with the community shows a proactive approach that aligns with the best interests of the dYdX ecosystem.

Imperator team

3 Likes

Thanks for the answer on the feasibility of bare-metal and thanks for continuing to run the indexer!

I hope at some point Trading can change the architecture because raking up a yearly 2m USD AWS bill is rough.

2 Likes

The dYdX Operations SubDAO will publish their on-chain proposal for the dYdX Community to commit to the above detailed 3.0 Mandate next Monday September 30th to not interfere with the currently live proposals for the Treasury subDAO.

1 Like

We have now posted a text proposal here to seek dYdX Community Consent to the proposed 3.0 Mandate of the dYdX Operations subDAO.

If this proposal is successful we will create separate funding proposals at a later date, towards the end of 2024.

2 Likes

25% is around $2.2M for Imperator. This is like a third of the Core contributors total expenses, and more than other budget categories. How reasonable is this $2.2M bill? And more importantly, why was only Imperator selected for this? I mean $2.2M is probably more revenue than all annual validator commission from all the 60 dYdX active set validators combined, other validators should be given an opportunity also. I mean if we look for example at uptime in the last 30 days, Imperator missed 1587 blocks, we missed only 561. Maybe there was some initial agreement with Imperator but now we are talking about $2.2M bill/3 years deal, there should be a RFP for validators to apply and several validators evaluated, maybe Imperator will be chosen again or no, but the current setup seems very unfair.
There will be 9 FTE at the subDAO? so that would be around $200k annual salary on average, seems pretty high, for a comparison Zurich offers the highest salaries globally, and in Zurich Google offers some of the highest salaries which are around $200k. But Imperator would be earning a $730k annual salary

1 Like