Post-Mortem Analysis: Imperfect Price Feed Impacting Nolus Protocol

Date and Time of Occurrence: September 8th, 2023, 08:45 UTC

Incident Overview: An erroneous price feed culminated in the untimely liquidation of Lease Positions across 86 distinct wallets.

Background

In our bid to accommodate Osmosis’ Supercharged liquidity pool variations within the Nolus Protocol, it became evident that the extant GAMM module — integral to liquidity pool operations on the Osmosis DEX — lacked compatibility with the novel pool formats.

Following our query routed via the GAMM module [1] on their test-net, “osmo-test-5”, we encountered an error [2]. We sought guidance from the Osmosis team who provided us with an alternative URL [3] pointing us at the new PoolManager module.

It’s pertinent to note a couple of key observations at this juncture: The new URL [3] presented parameters, “base_asset_denom” and “quote_asset_denom”, in their original order, suggesting identical behavior with the preceding API. We didn’t pursue further testing under the assumption that everything remained consistent, leading to the challenges that arose later.

Core Issue Identification

The pivotal discord arises from the PoolManager module’s inverse price representation relative to the GAMM module, given the same query parameters. For instance, a GAMM query for a pool containing 50 ABC to 100 XYZ (with ABC as the base and XYZ as the quote) returns a price of “0.5”. Conversely, the PoolManager module, under identical conditions, returns “2.0”. The PoolManager’s data, for instance, “0.000039”, when integrated into GAMM-tuned contracts, would inadvertently imply “0.000039 USDC is valued at 1 WBTC” or otherwise said “1 USDC is valued at 25784 WBTC”.

Mitigative Steps

1. Immediate Response: The price feeds were promptly reverted to the GAMM module, arresting further unwarranted liquidations spurred by price disparities.
2. Enhanced Validation Protocols: Our software validation protocol has been significantly augmented. Beyond automated testing and peer reviews, it now encompasses multi-level manual quality assurance checks within staged deployments, guaranteeing the highest standards of performance and reliability prior to transitioning into the production environment.
3. Robust Price Feed Mechanisms: To bolster our system’s resilience, all price feeds will amalgamate data from additional sources and cross-check for accuracy before being supplied to the Nolus Protocol oracle contract. If the benchmark data significantly deviates from the figures returned by Osmosis, those feeds will be withheld from the protocol, concurrently activating a series of alerts. This empowers our core contributors to quickly address the discrepancies, ensuring uninterrupted system performance.

At Nolus, our dedication to crafting high-quality code is paralleled only by our commitment to ensuring robust and reliable solutions. We understand the critical nature of every line we write, and we aim to set a gold standard in software excellence. While the digital landscape is ever-evolving and challenges are inevitable, we are steadfast in our promise to minimize incidents.

Our past has taught us, our present motivates us, and our future is about continually raising the bar, striving for impeccable performance with fewer disruptions.

[2] GAMM module JSON return

{
  "code": 2,
  "message": "rpc error: code = Internal desc = pool with ID 146 does not exist: unknown request",
  "details": []
}

Sources

• [1] URL, GAMM query: Supercharged pool — lcd.osmotest5.osmosis.zone/osmosis/gamm/v1beta1/pools/146/prices
• [3] PoolManager module URL — lcd.osmotest5.osmosis.zone/osmosis/poolmanager/pools/146/prices