Every gap, and why.

Not established. What is established: Deribit closed our chain websocket connections without a close frame at 17:51:49Z, and eight seconds later began answering HTTP 429 Too Many Requests to websocket handshakes and REST calls alike, in all three collectors that talk to it, until 18:01:29Z. Capture resumed one second after the last refusal. What is not established is why the exchange refused us. That matters because this is the first incident since the reconnect fix of 13 October, and on its face it has the shape that fix was written for: the ten-minute, seven-currency hole of 1, 6 and 8 October. But the fix was running, and it did what it was built to do. The drop was read as a cohort of seven, each connection drew its own extra delay across a 10.5-second spread, and the pool re-subscribed 831 channels over about four seconds, where on 8 October it re-subscribed 1,418 inside one. The refusal came anyway. Two readings were recorded first, and nothing in the record separates them. Either a burst that size, four seconds wide, is still enough to trip the exchange's limit, and the fix narrowed the herd without getting it under the threshold; or the refusal was the exchange's own, arriving with the disconnection rather than caused by our return. The futures collector points weakly toward the second: it lost no connection and re-subscribed nothing, yet it was refused 539 times in the same window. Only weakly, because the futures collector was refused in the earlier herd incidents too, so a shared limit tripped by the chain collectors would look the same from here. A third reading was found on 18 October, and it is the only one of the three with a measurement behind it. Deribit allows 32 simultaneous connections from one address, REST requests and websockets counted together, and refuses the 33rd with 429. On 18 October our collectors held 27, thirteen of them the chain sockets of the collector whose connections dropped here. A connection closed without a close frame may go on counting at the exchange until it times out. If it does, reopening thirteen needs thirteen places with five free, and every new request from every collector is refused until the dead ones expire. That would account for what the first two readings strain at: spreading the reconnects over ten seconds made no difference, the futures collector was refused without reconnecting anything, and the book daemon, which needed nothing new, was never refused. It is still a reading. Whether the exchange counts a dead connection is not something we can see from our side. On 18 October the chain collectors were changed to carry the same channels on five sockets instead of twenty-one, which took the address from 27 connections to 12. The next mass disconnection tests it: one that passes without a refusal supports this reading, and a refusal anyway rules it out. Until then we record unknown rather than pick one. Two things differ from 1 October and are worth stating. There was no flapping beforehand: not one disconnection in the twenty minutes before the drop. And the L2 book daemon was untouched: its connection was neither dropped nor refused, it did not restart, and it wrote BTC and ETH order-book updates throughout with no interval over two seconds, so no order-book data was lost. The chain snapshots for the ten minutes are a true hole, for the same reason as the rest of this record's holes. Detection and notification both worked: health_check alerted four minutes in, and again on recovery.

evidence - All five collector units active with NRestarts=0 since 2026-10-14 07:08Z, which is after the reconnect dispersion fix of 13 October was deployed; its log event, deribit_ws_reconnect_dispersed, appears 56 times in volar-collector and 29 times in volar-collector-altcoins in the window. Mass disconnection at 17:51:49Z, every drop carrying error='no close frame received or sent', on connections that had been held between 2.4 and 8.2 hours. Zero disconnections in either chain collector between 17:30Z and 17:51:48Z. Dispersion as logged: cohort=7 with spread=10.5 and cohort=6 with spread=9.0 in volar-collector, extra delays including 0.289s, 2.601s and 8.413s; cohort=2 with spread=3.0 in volar-collector-altcoins. Re-subscriptions before the first refusal: deribit_ws_subscribed count=200 at 17:51:50Z and three more at 17:51:53Z in volar-collector, count=27 and count=4 at 17:51:50Z in volar-collector-altcoins, 831 channels in all. First refusal 17:51:57Z, a websocket handshake rejected with 'server rejected WebSocket connection: HTTP 429'. Handshake refusals: 49 in volar-collector (17:51:57Z to 18:01:16Z), 27 in volar-collector-altcoins (17:54:36Z to 18:01:11Z). REST refusals as deribit_request_retry with '429 Too Many Requests': 87 in volar-collector (17:52:29Z to 18:01:29Z), 144 in volar-collector-altcoins (17:52:30Z to 18:00:53Z), 539 in volar-collector-futures (17:52:48Z to 18:01:29Z), the last of these with zero websocket disconnections and zero re-subscriptions in the window. Zero 503 responses anywhere. Last snapshots before the hole 17:51:29Z to 17:51:33Z, first after 18:01:30Z to 18:01:33Z, identical across all seven currencies, with one snapshot per currency per minute either side and nothing in between. No other process on the box logged a Deribit request in the window. volar-collector-book: no disconnection, no refusal, NRestarts=0; book_write for BTC and for ETH 656 times each between 17:51Z and 18:02Z, longest interval between writes 2.0 seconds across 17:40Z to 18:10Z. health_check.log: 'Chain stale (threshold=120s)' alerted at 17:55:02Z with all seven currencies at 211 to 212s, repeat suppressed at 18:00:03Z at 512s, 'Chain capture has recovered' at 18:05:02Z, and no delivery failure logged. Measured on 2026-10-18 on the production host (ss, established outbound HTTPS): 27 connections from the collector processes before the change, 12 after it, made up of volar-collector 5, volar-collector-altcoins 4 and one each for the book, futures and funding collectors. Ticker sockets before, at 200 channels each: 13 in volar-collector and 8 in volar-collector-altcoins. After, as the collectors logged on restart: connections=3 for 2,464 channels and connections=2 for 1,554. The book daemon carries 1,722 channels on one socket, so the 200-per-socket figure was never an exchange limit. The 32-connection limit is Deribit's published connection policy (Connection Management Best Practices, support.deribit.com).

Deribit returned HTTP 503 Service Unavailable to everything we asked of it (REST calls and websocket handshakes alike) for forty-one minutes. Capture resumed 57 seconds after the last refusal. The start time, 09:00Z, matches a long row of earlier interruptions on this record, every one of them confirmed as an exchange outage. Our reconnection behaviour did not extend it, and this is the third incident where that can be shown rather than assumed: the websocket pool dropped 816 times and drew 816 distinct delays, which is the jitter added on 14 September doing its job. The sharper evidence is again an absence: not one 429 response in the window. A synchronised return would have produced them, as it did on 22 September. What was ours, and is not fixed by this entry: the L2 book daemon does not survive an outage it cannot reach through. It exhausted its reconnect budget and exited on 'server rejected WebSocket connection: HTTP 503', systemd restarted it, the new process met the same 503, and that cycle ran thirteen times across the window until the exchange came back. The chain collectors, which retry in place and never exited, were in exactly the same position and lost exactly the same data, so the restart loop neither helped nor hurt the outcome here. It is recorded because 2026-10-01 had the identical shape, which makes it twice, and because an exchange outage long enough to exhaust the budget is the normal case rather than the exceptional one. The forty-one minutes of order-book deltas inside the window are lost and unrecoverable; there is no historical book endpoint to refill them from. The chain snapshots are a true hole for the same reason the rest of this record's holes are. One further thing this incident exposed about our own instrumentation, now being fixed separately: every failed cycle logged a full traceback with locals, about 1,240 journal lines a minute against a baseline of 15, roughly 50,000 lines across the window. The traceback renders our own source, so a grep for '429' in the window returns 254 hits: all of them one comment line in our retry code, and none of them an exchange response. Instrumentation that buries the signal under its own source listing makes the next diagnosis slower, not faster.

evidence - 2,052 responses carrying '503 Service Unavailable' in the primary chain collector between 09:01:04Z and 09:40:08Z, sustained at about 54 per minute with no break; 2,880 in the altcoin collector and 2,590 in the futures collector over the same window, with 456 and 640 deribit_request_retry events behind them. Zero '429 Too Many Requests' responses in any collector. Last chain writes before the hole 09:00:05Z (BTC), 09:00:06Z (ETH), 09:00:07Z (SOL); first writes after 09:41:05Z, 09:41:06Z, 09:41:07Z, 2,460 seconds, identical across all seven currencies. Last 503 at 09:40:08Z, first assembled snapshot 09:41:05Z. 816 deribit_ws_disconnected events paired with 816 distinct reconnect_delay values. The chain, altcoin, futures and funding units all stayed active throughout with NRestarts=0, up since 2026-10-12 13:58 CEST. volar-collector-book is the exception: 13 occurrences of 'InvalidStatus: server rejected WebSocket connection: HTTP 503' with the reconnect budget exhausted each time, matching NRestarts=13 (Restart=always, RestartSec=5s), the first at 09:01:18Z and the last at 09:36:36Z, with the surviving process active from 09:41:36Z. Detection and notification both worked: 'Chain stale (threshold=120s)' alerted at 09:05:02Z with all seven currencies at 296s, repeats suppressed through 09:40:03Z at 2,397s, 'Chain capture has recovered' at 09:45:02Z, and all of it delivered to a human; the alert channel had been repaired earlier the same day.

The third instance in eight days of the failure described in the 1 October entry, and the cleanest one: nothing else was wrong at the time, so the mechanism stands on its own. Deribit dropped nine connections at 01:34:15Z. One second later every one of them had reconnected and re-subscribed (1,418 channels inside that second) and the first handshake rejection is stamped in the same second as the re-subscribe. Capture stopped for all seven currencies for the next nine minutes and resumed at 01:44Z, with no further drop in between. Unlike 1 and 6 October the pool was not flapping beforehand: three drops in volar-collector across the preceding twenty minutes, against 96 before the 1 October incident. One ordinary mass disconnection, answered by our entire pool at once, is sufficient on its own to cost ten minutes of capture across every currency we sell. The L2 book daemon is the control that isolates the cost to our own re-subscription. It held an established connection throughout, needed no new request, and was never refused: 3,577 writes in the window, zero disconnections, zero 429s, while every process that had to ask Deribit for something new was being turned away. The exchange was not unwell: it was refusing us, and specifically refusing the new requests we had just made far too many of. Our own steady-state rate was unchanged going in, which is the remaining open question rather than a defence: it shows nothing on our side spiked before the drop, not that our ordinary request rate sits comfortably inside the budget. The incidents of 10 and 11 September reached the same wall from the same baseline.

evidence - Mass disconnection at 01:34:15Z: 3 drops logging attempt=0 in volar-collector and 6 in volar-collector-altcoins, every one carrying error='no close frame received or sent'. reconnect_delay for those nine drops: 0.570, 0.736, 0.775 (chain) and 0.690, 0.703, 0.732, 0.838, 0.876, 0.877 (altcoins): the whole pool inside a 0.31-second spread of the attempt-0 interval. At 01:34:16Z: deribit_ws_connected and deribit_ws_subscribed across both units totalling 1,418 channels (600 in volar-collector as 3 × count=200; 818 in volar-collector-altcoins as 4 × count=200 plus count=18), and in that same second volar-collector-altcoins logged attempt=1 with error='server rejected WebSocket connection'. First REST refusal 01:34:28Z. 326 deribit_request_retry warnings carrying "429 Too Many Requests": 108 in volar-collector (01:35:08Z → 01:43:25Z), 120 in volar-collector-altcoins (01:35:07Z → 01:43:31Z), 98 in volar-collector-futures (01:34:28Z → 01:43:03Z), on /public/get_instruments and /public/get_book_summary_by_currency. Zero 503 responses in the window. Per-currency write minutes 01:28Z → 01:49Z: all seven continuous to 01:34, nothing from 01:35 to 01:43, all seven resuming together at 01:44. Request rate before the event flat and ordinary (3 snapshots a minute in volar-collector, 4 in volar-collector-altcoins, matching the 9 October control day) and no scheduled job overlaps the window. volar-collector-book across the same window: 3,577 book_write events and no other event type at all; zero deribit_ws_disconnected, zero 429s, no systemd restart. Pre-window churn, 01:14Z → 01:35Z: 3 disconnections in volar-collector and 11 in volar-collector-altcoins, against 96 and 61 in the equivalent window on 1 October. health_check detected it at 01:40:03Z ('Chain stale (threshold=120s)', all seven at 354s) and recorded recovery at 01:45:03Z; the gap monitor raised 'Capture gap with no recorded cause' at 04:05:04Z. No alert email worked until 13 October.

The same failure as 1 October, five days later and from the same point in the code: Deribit dropped the websocket pool, the pool returned inside one second and re-subscribed 1,107 channels at once, and was then refused with 429 for nine minutes. The mechanism, and why none of the September fixes prevent it, is set out in the 1 October entry. What this incident adds is a warning about the gap list itself. The coverage report names four currencies (BTC, ETH, HYPE and XRP) and that understates what happened. All seven stopped after 15:07Z and none wrote again until 15:16Z. SOL and AVAX each landed a single snapshot at 15:12Z and TRX one at 15:11Z, which split their absence into two runs each shorter than the detector's threshold; they are therefore missing from the published scope while having lost almost exactly as much data as the four that are named. health_check's own freshness numbers four minutes in say it plainly: BTC 441s and ETH 441s against SOL 201s and AVAX 199s, which is one lucky write of difference. A single write inside a hole hides the hole. That is a defect in what we publish rather than in this incident, and it is recorded here because this is where it was found. The L2 book daemon again exhausted its reconnect budget and was restart-looped by systemd through the refusal, costing 912 seconds of BTC and ETH order-book deltas that cannot be recovered.

evidence - Websocket churn from 15:06:28Z: 4 drops logging attempt=0 in volar-collector and 2 in volar-collector-altcoins, error='no close frame received or sent', with reconnect_delay between 0.567 and 0.9997: all inside the attempt-0 interval. 1,107 channels re-subscribed (707 and 400). First refusal 15:08:03Z as a rejected handshake; the L2 book daemon took the same rejection at 15:08:11Z. 321 deribit_request_retry warnings carrying "429 Too Many Requests": 103 in volar-collector (15:08:40Z → 15:17:43Z), 120 in volar-collector-altcoins (15:08:42Z → 15:17:43Z), 98 in volar-collector-futures (15:08:51Z → 15:17:27Z). Zero 503 responses in the window. Per-currency write minutes, 14:58Z → 15:24Z. BTC, ETH, HYPE and XRP: continuous to 15:07, then nothing until 15:16 (ETH, HYPE, XRP) and 15:17 (BTC). SOL: 15:07, one write at 15:12, then 15:16. AVAX: 15:07, one at 15:12, then 15:16. TRX: 15:07, one at 15:11, then 15:16. health_check at 15:15:03Z: 'AVAX=199s BTC=441s ETH=441s HYPE=439s SOL=201s TRX=199s XRP=439s'. Pre-window flapping, 14:48Z → 15:09Z: 49 disconnections and 40 reconnections in volar-collector, 56 and 47 in volar-collector-altcoins. volar-collector-book: 7 × 'InvalidStatus: server rejected WebSocket connection: HTTP 429', 7 × book_daemon_max_reconnects_exhausted, systemd restart counter reaching 5 by 15:10:41Z. health_check reported book write intervals BTC=912s ETH=912s at 15:25:04Z.

Deribit closed our websocket pool without a close frame. That alone is routine and recoverable. What turned it into the gap published here is that the whole pool came back in the same second and re-subscribed every channel at once; Deribit then answered 429 Too Many Requests to everything that needed a new request (REST snapshot calls and websocket handshakes alike) and all seven currencies stopped. This is the fourth incident in the family that begins on 2026-07-14, and the first of three inside eight days (see 6 and 8 October). The mechanism is now measured rather than inferred. The backoff counter resets for any connection that held sixty seconds, so after a mass drop almost every connection reconnects at attempt 0, and at attempt 0 the jittered delay is drawn from [0.5, 1.0] seconds. A pool dropped together therefore returns together: twenty-three connections reconnected within four seconds of the drop and re-subscribed 1,600 channels, and the first refusal arrived one second after the drop. The pool had also been flapping for the preceding twenty minutes, 157 disconnections across the two chain collectors, each drop manufacturing a fresh attempt-0 herd. None of the four fixes made in September reach this. Honouring Retry-After (13 August) governs how we retry a refusal, not how we return from a disconnection. Per-connection subscribe pacing (10 September) spreads two hundred channels inside one connection and cannot spread thirteen connections. Half jitter (14 September) randomises a fraction of the backoff ceiling, and at attempt 0 that ceiling is one second, so it disperses the herd across half a second. Reset-on-survival (1 September) decides when the counter returns to zero, and makes attempt 0 the common case rather than the rare one. The herd forms at the one point none of them act on, which is why the same ten-minute outage appears three more times on this record. The L2 book daemon failed louder in the same refusal: rejected on the handshake, it exhausted its reconnect budget, exited, and systemd restarted it into the same rejection, the shape repeated on 13 October. 617 seconds of BTC and ETH order-book deltas inside the window are lost and unrecoverable, because there is no historical book endpoint to refill them from. The chain snapshots are a true hole for the same reason as the rest of this record's holes. Nobody was told while it was happening. health_check detected it at 06:05:03Z and wrote 'Chain stale (threshold=120s)' for all seven currencies to its log, but alert email did not work until 13 October, so none of these three incidents reached a human at the time.

evidence - Mass disconnection at 05:59:04Z: 14 drops logging attempt=0 in volar-collector and 9 in volar-collector-altcoins within four seconds, every one carrying error='no close frame received or sent'. The reconnect_delay values recorded for those drops run from 0.565 to 0.999: the [0.5, 1.0] interval _backoff_delay() returns at attempt 0, where ceiling = min(2**0, max_backoff) = 1 (volar/collector/deribit.py:807). 1,600 channels re-subscribed in the same window, as deribit_ws_subscribed count=200 eight times over. First refusal 05:59:05Z, a websocket handshake rejected with 'server rejected WebSocket connection: HTTP 429'; the L2 book daemon took the same rejection at 05:59:07Z. 337 deribit_request_retry warnings carrying "429 Too Many Requests": 102 in volar-collector (05:59:26Z → 06:08:42Z), 141 in volar-collector-altcoins (05:59:25Z → 06:08:44Z), 94 in volar-collector-futures (05:59:33Z → 06:08:02Z). Zero 503 responses anywhere in the window: the exchange was serving other traffic and refusing ours. Last chain write before the gap at 05:58Z for all seven currencies; first write after it 06:08Z (HYPE, AVAX) and 06:09Z (BTC, ETH, SOL, TRX, XRP), with no missing minute either side. Our own request rate was unchanged across the event: 3 snapshots a minute in volar-collector and 4 in volar-collector-altcoins for the hour before, identical to the 9 October control day. No other process on the box issued a Deribit request in the window, and no scheduled job overlaps it; the nearest are the backfills at 00:00Z, 00:30Z and 00:45Z. Pre-window flapping, 05:39Z → 06:00Z: 96 disconnections and 82 reconnections in volar-collector, 61 and 52 in volar-collector-altcoins. volar-collector-book: 7 × 'InvalidStatus: server rejected WebSocket connection: HTTP 429', 7 × book_daemon_max_reconnects_exhausted, systemd restart counter climbing from 1 at 05:59:12Z. health_check reported book write intervals BTC=617s ETH=617s at 06:10:03Z.

Deribit refused every websocket connection for seventeen minutes, answering HTTP 503 to the handshake itself rather than dropping established sessions. Capture resumed thirteen seconds after the last refusal. Our own reconnection did not extend it, and this is the second incident where that can be shown rather than assumed. The pool reconnected 413 times across the window and drew 413 distinct delays; the jitter added on 14 September is what produces that. Under the previous behaviour every connection computed the same delay and returned together, which is what turned two exchange hiccups in September into ten-minute self-inflicted outages. The sharper evidence is the absence: zero 429 responses in the window. A synchronised return would have produced them, as it did on 22 September when an unattended database restart put every collector back in lockstep and 387 of them arrived at once. Here the exchange was refusing everything and there was nothing to capture regardless.

evidence - 1,519 responses carrying 503 between 09:00:38Z and 09:17:25Z, against the websocket handshake ('server rejected WebSocket connection: HTTP 503') rather than against REST. Last snapshot before the hole 08:59:38Z, first after 09:17:38Z, 1,080 seconds, identical across all seven currencies. 413 deribit_ws_disconnected events paired with 413 distinct reconnect_delay values spanning 0.51s to 59.99s. Zero 429 responses and zero deribit_rest_slow events in the window. All seven collector units stayed active throughout; no unit restarted.

An unattended package upgrade restarted the database underneath the collectors, and the synchronised recovery that followed hit the exchange's rate limit. apt-daily-upgrade.service runs from a systemd timer at 06:00 local every day. On this run it stopped and started postgresql@16-main. All five collector units lost their database connections and restarted within twelve seconds of each other, which left every Deribit websocket in the pool opened at the same moment. Sixteen minutes later those connections dropped together, reconnected together, and each one issued its post-reconnect instrument refresh at the same time. Deribit answered 429 Too Many Requests, and REST is what a snapshot needs for open interest and volume, so no snapshot could be assembled until the throttle cleared eight minutes later. The exchange was healthy throughout: zero 503 responses in the window. The 429s were a refusal of our own request burst, not an outage. Worth separating two things that look alike. The reconnect jitter added on 14 September worked here: three disconnections drew three distinct delays, so the websockets did not return in lockstep. What is not jittered is the REST call each connection makes after it reconnects: get_instruments and get_book_summary_by_currency fire immediately on recovery, and it was those, not the reconnections, that bunched. The fix for the websocket herd did not cover the REST herd behind it. The trigger recurs daily and is not specific to this date.

evidence - Last snapshot before the hole 04:27:33Z, first after 04:35:33Z, 480 seconds, identical across all seven currencies. systemd: 'Starting apt-daily-upgrade.service' 04:10:42Z, 'Stopping postgresql@16-main' 04:10:46Z, 'Started postgresql@16-main' 04:10:52Z, unit finished 04:11:13Z. Collector unit start times that follow: book/futures/funding 04:10:57Z, altcoins 04:10:59Z, chain 04:11:09Z. 387 deribit_request_retry events against 429 Too Many Requests between 04:27Z and 04:36Z, on get_instruments (BTC, ETH, USDC) and get_book_summary_by_currency. 3 deribit_ws_disconnected events at 04:27:53Z against 0 in a comparable two-minute window at 05:00Z, paired with 3 distinct reconnect_delay values. 0 responses carrying 503 in the window. apt-daily-upgrade.timer OnCalendar=*-*-* 06:00:00, next elapse 2026-09-23 06:00 local.

Deribit returned HTTP 503 Service Unavailable continuously for forty minutes. Capture resumed within a minute of the last refusal. The start time, 08:59Z, matches eight earlier interruptions on this record, all confirmed as exchange outages. Our own reconnection behaviour did not extend it, and this is the first incident where that can be shown rather than assumed. The websocket dropped 719 times across the window and every one of those reconnections drew a different delay: 719 disconnections, 719 distinct values. The jitter added on 14 September after the 10 and 11 September interruptions is what produces that. Under the previous behaviour every connection computed an identical delay and they returned together, which is what turned two exchange hiccups into ten-minute self-inflicted outages. Here the exchange was refusing everything for forty minutes and there was nothing to capture regardless.

evidence - 3,148 x 503 responses in the window, sustained at roughly 85 per minute from 09:00Z through 09:38Z with no break. Last 503 at 09:38Z; first snapshot written 09:39:13Z, 40 minutes after the 08:59:12Z snapshot that preceded the hole. 444 deribit_request_retry events against 429 responses, secondary to the outage, a throttle on retries while the exchange was already failing. 719 deribit_ws_disconnected events paired with 719 distinct reconnect_delay values.

The same failure as the day before, with the fix for it already deployed. Deribit throttled us with 429 for ten minutes and every currency stopped. 96 retries against /public/get_book_summary_by_currency and /public/get_instruments, our ordinary calls, refused, because the request budget had gone elsewhere. The fix shipped on 10 September paced each connection's re-subscription. It does nothing here, and the reason is worth recording. The ticker pool spreads instruments across about thirteen websocket connections holding two hundred channels each, so no single connection ever sends a large subscribe. When Deribit drops the connections it drops all of them together, they back off on identical schedules, and they reconnect in the same instant: eight of them waited exactly one second and then re-subscribed at once. Around two thousand six hundred subscriptions arrive simultaneously, and pacing within one connection cannot spread a burst that is spread across thirteen. This is a thundering herd, and the missing piece is jitter: the backoff delays are identical because nothing randomises them. That is the change being made now. The 10 September entry's account of the mechanism was wrong on this point and has been left as written rather than quietly corrected, because the sequence of two wrong readings is itself part of what happened.

evidence - 96 deribit_request_retry events between 07:44Z and 07:54Z: 54 against /public/get_book_summary_by_currency and 42 against /public/get_instruments. Only 3 lines mention 503, so this was throttling rather than an exchange outage. 22 websocket disconnections and 8 completed re-subscriptions, each logged count=200 messages=1, confirming that no individual connection sent a large subscribe. Reconnect delays in the window were 1s x8, then 2s, 4s, 8s, 16s, 32s once each, then 60s x9; eight connections sharing one delay is the herd. Capture stopped after the 07:44:21Z snapshot and resumed at 07:54:21Z, a 600-second hole across all seven currencies.

We exhausted our request budget at Deribit and were throttled with HTTP 429 for ten minutes. There were no 503 responses, so the exchange was serving other traffic and refusing only ours. It was not caused by polling too hard. Our routine REST rate is about seven requests a minute (one book summary per currency, plus an instrument list per currency every five minutes) and during the interruption almost every one of those ordinary calls was refused. The budget had been spent elsewhere: each time the websocket reconnects, the collector re-subscribes to a ticker channel for every live instrument, 2,440 of them. Nine reconnections completed inside those ten minutes, so roughly twenty-two thousand subscription operations landed in a limit that REST and websocket traffic share, and the ordinary calls had nothing left to spend. This is the fourth interruption of this family, after 2026-07-14, 2026-08-13 and 2026-09-01, and the first where the cause is the cost of reconnecting rather than the behaviour of a retry. The two earlier fixes both held: the REST path honoured Retry-After, and the reconnect backoff escalated 1s, 2s, 4s, 8s, 16s, 32s to a 60-second ceiling, so 65 attempts produced only nine completed reconnections where the same shape on 1 September produced two hundred. Without that the interruption would have been longer. What is not yet fixed is the price of a reconnection. Re-subscribing every instrument at once is affordable occasionally and not affordable nine times in ten minutes.

evidence - 108 deribit_request_retry events between 14:09Z and 14:17Z, an even 12 per minute, being 54 distinct requests each retried twice: 27 against /public/get_instruments and 27 against /public/get_book_summary_by_currency. Zero 503 responses. (A raw grep for '429' in the journal returns 761 lines; 653 of those are rich-traceback decoration around the same 108 events, and the larger figure is not a request count.) 65 x deribit_ws_disconnected against 9 x deribit_ws_subscribed, each subscribe covering 2,440 instrument channels; the first 429 landed in the same second as the first resubscribe, 14:08:50Z. Capture stopped after the 14:08:32Z snapshot and resumed at 14:18:32Z, a 600-second hole across all seven currencies. No service restarted; volar-collector has NRestarts=0 and has been up since 7 September.

Deribit returned HTTP 503 Service Unavailable continuously for the whole window. Capture resumed within a minute of the last refusal. The L2 book daemon separately exhausted its reconnect budget and was restarted by systemd 270 times during the same period; that is our defect and is being fixed, but it did not lengthen this interruption: the exchange was refusing every request throughout, so there was nothing to capture, and chain collection resumed as soon as the 503s stopped.

evidence - 3,456 x 503 responses in the window, sustained at roughly 87 per minute from 09:00Z to 09:35Z with no gap, plus 204 x 429 and 5 x 403. Last 503 at 09:35Z; first snapshot written 09:36Z. The book daemon logged book_daemon_max_reconnects_exhausted and recovered at 09:36:17Z with book_subscriptions_refreshed count=1826.

Deribit closed every one of our websocket connections at the same second, without a close frame. That alone is routine and recoverable. What turned it into a ten-minute interruption is our reconnect loop: the backoff counter resets on a successful CONNECT rather than on a connection that survives, so a socket that establishes and immediately dies restarts the delay at one second indefinitely. It never escalated. Two hundred reconnect cycles followed, each re-subscribing every channel, and Deribit answered 820 times with 429 Too Many Requests. Capture resumed when the cycle settled. This is the third rate-limit incident of the same family after 2026-07-14 and 2026-08-13; the fix made in August taught the REST retry path to honour Retry-After and did not touch the websocket reconnect path, which is where this one happened.

evidence - 820 x 429 Too Many Requests and no 503 responses at all, so the exchange was serving other traffic and refusing ours. 200 x deribit_ws_disconnected paired with 200 x deribit_ws_connecting, every one logged attempt=0, which is the reset. First disconnections all share the timestamp 06:01:37Z with error 'no close frame received or sent'.

Deribit returned HTTP 503 Service Unavailable.

evidence - 1,205 x 503 responses in the window, alongside 85 x 429.

Deribit returned HTTP 503 Service Unavailable. The longest interruption on record.

evidence - 6,666 x 503 responses in the window, alongside 516 x 429.

Observed at the time, but the cause is not established. A second degraded window the same afternoon, ten minutes after the first, on ETH and SOL but not BTC. Same reason as the 14:00 window: no journal survives from this date.

evidence - [2026-08-16T14:20:15Z] health_check.log: 'Chain stale (threshold=120s): ETH=162s SOL=162s', naming exactly the two bases the measurement scopes this window to, and not BTC, from an independent source. Measured from chain_snapshots: 14:17:19Z to 14:23:48Z, intervals of 208 s and 181 s around a single write. Seven staleness alerts that day against 0 the day before.

Observed at the time, but the cause is not established. Capture degraded on all three bases then collected for about seven minutes. The journal no longer reaches back past 2026-09-04, and this window predates it by five weeks.

evidence - [2026-08-16T14:05:02Z] health_check.log: 'Chain stale (threshold=120s): BTC=220s ETH=220s SOL=270s', inside the window measured from chain_snapshots (SOL 14:00:31Z-14:07:19Z, BTC and ETH 14:01:22Z-14:07:19Z; two consecutive intervals each over twice the day's median, neither over five minutes on its own). SOL's larger staleness in the alert matches its earlier measured start.

We exceeded Deribit's request rate limit again and were throttled with HTTP 429. This is a recurrence of 2026-07-14, from the same line of code: the retry path waited 0.1s, 0.2s, 0.4s for every error including a throttle, and ignored the Retry-After header telling us how long to wait. Retrying a rate limit after 100ms is what keeps it tripped, so our own traffic sustained the outage a second time. The July incident named this defect and nothing was changed; publishing a cause is not the same as fixing it. The retry path now honours Retry-After, with a one-second floor and a sixty-second ceiling, for 429 only: a 503 means the exchange is down rather than annoyed, and still retries fast.

evidence - 306 × 429 Too Many Requests in the window, with 136 websocket reconnects. No service restart.

Deribit returned HTTP 503 Service Unavailable.

evidence - 1,566 × 503 responses in the window.

Observed at the time, but the cause is not established. All three bases then collected went stale together for about four minutes, inside a day that was already degraded end to end: 2026-08-06 ran at a 92.7 s median against a normal 80-84 s and produced 894 ETH snapshots instead of ~1055, as roughly ten seconds added to every interval. Whether the exchange was slow or we were, nothing surviving can say: the journal no longer reaches back past 2026-09-04.

evidence - health_check.log, which samples every five minutes: OK at 00:45:02Z, then [2026-08-06T00:50:13Z] 'Chain stale (threshold=120s): BTC=247s ETH=247s SOL=247s', then OK at 00:55:02Z, bracketing the window measured from chain_snapshots (00:45:55Z to 00:54:24Z, intervals of 278 s and 231 s either side of a single write). 27 staleness alerts that day, against 0 the day before and 2 the day after. The identical 247 s on every base points to one shared cause rather than three.

Deribit returned HTTP 503 Service Unavailable.

evidence - 1,053 × 503 responses in the window.

Deribit returned HTTP 503 Service Unavailable.

evidence - 459 × 503 responses in the window.

We exceeded Deribit's request rate limit and were throttled with HTTP 429. The collector's retry path had no backoff proportional to the throttling, so it kept requesting and kept being refused: our own traffic sustained the outage. Capture recovered and failed again between retries, so it is measured as two separate interruptions rather than one continuous outage.

evidence - 510 × 429 Too Many Requests, and 300 reconnect attempts, in the window.

We exceeded Deribit's request rate limit and were throttled with HTTP 429. The collector's retry path had no backoff proportional to the throttling, so it kept requesting and kept being refused: our own traffic sustained the outage. Capture recovered and failed again between retries, so it is measured as two separate interruptions rather than one continuous outage.

evidence - 510 × 429 Too Many Requests, and 300 reconnect attempts, in the window.

Deribit returned HTTP 503 Service Unavailable. Longest to date.

evidence - 4,347 × 503 responses in the window.

Deribit returned HTTP 503 Service Unavailable.

evidence - 1,107 × 503 responses in the window.

Deribit rejected requests with HTTP 403 Forbidden. The rejection was at the edge rather than the API, and cleared without action from us.

evidence - 126 × 403 responses in the window.

Deribit returned HTTP 503 Service Unavailable.

evidence - 378 × 503 responses in the window.

Deribit returned HTTP 503 Service Unavailable to every request.

evidence - 1,674 × 503 responses on /public/get_instruments in the window.

Not established.

evidence - Falls two hours before the retained journal begins (2026-05-13T15:53Z).

Not established.

evidence - Predates collector log retention.

The cause is not established and cannot now be established. An eight-minute degraded window on all three bases then collected, newly visible on 2026-10-14 when the gap rule began counting consecutive degraded intervals rather than single ones. Nothing we still hold covers it: collector logs reach back to 2026-05-13T15:53Z, a day later, and the health-check log's first dated line is 2026-05-21T22:55Z. It resembles later confirmed incidents, and a matching signature is not evidence.

evidence - Measured from chain_snapshots: consecutive intervals over twice the day's median (73 s) spanning 12:51 to 12:59 on all three bases then collected, totalling over five minutes with no single interval over five minutes, which is why the previous rule reported nothing here. No contemporaneous log of any kind exists for this date.

Not established. The start time matches six later interruptions that were confirmed as exchange outages, but a matching signature is not evidence and this one is not claimed as upstream.

evidence - Predates collector log retention.

Not established.

evidence - Predates collector log retention.

Not established.

evidence - Predates collector log retention.

Not established.

evidence - Predates collector log retention (logs begin 2026-05-13T15:53Z).

The six-hourly job that regenerates the public evidence artifacts was a bare command in the crontab and did not source the service environment, so it had no database or object-store credentials. Rather than failing, it measured nothing and published an empty coverage report over a complete one. The public coverage page showed no coverage for fourteen hours.

Impact. Display only. Capture ran normally throughout (842 BTC snapshots at a 60-second median with no interruption over five minutes) and no data was lost or altered. One visitor loaded the page during the window.

Fix. The job refuses to start when a source is unreachable, refuses to write a report that measured nothing, and leaves the previous artifact serving. It now runs through a wrapper that loads the environment, as every other scheduled job already did.

36 served BTC chain files inside the declared gap window (2024-10-02..2026-04-20) carry source='tardis_monthly': 11,494,762 rows of third-party vendor data. That literal is not in SOURCE_VALUES at all, so validate_snapshot rejects it: the files were written by a path that skipped validation and published by one that never asked. purge_gap_chains.py was run with --apply on 2026-07-02 against exactly these 36 keys. It wrote a manifest recording all 36 as removed and printed 'Deleted 36 object(s) ... is now a true hole'. Every one is still there, with the row counts the manifest recorded. The delete was issued with Quiet=True, which asks S3 to return ONLY the failures, and the response was discarded. The single signal that could have contradicted that sentence was requested and thrown away, and nothing re-read the bucket afterwards. Why the delete did not take effect is not established: the run's output was not captured to a log, and it is not recoverable now.

Impact. /methodology describes 2024-10-02..2026-04-20 as modeled coverage (source=modeled_surface) and the landing page describes the same window as a genuine hole. For 36 days inside it, Pro customers could download undisclosed vendor data instead. The files also lack bid_price and ask_price, which breaks concatenation across the archive: the symptom the purge script's own docstring names. No customer is known to have taken one: file_downloads records 1,646 chain downloads and none of these 36 keys. The exposure was availability, not delivery. Separately and from the same audit: 234 of 249 served trade-tape files carry 15 of the 21 columns TRADE_TAPE_SCHEMA declares, so combo_id, block_trade_id, liquidation and contracts are absent from months /coverage lists as available from November 2022, returned as 200 with the columns missing rather than 404. And ~5% of served chain files still carry the internal `vendor` column, which is NULL in all of them, so nothing was disclosed by it.

Fix. The gate moved to the upload boundary. _upload_stripped already reads every chain Parquet to strip INTERNAL_COLUMNS, so it now refuses any file whose source values are absent from SOURCE_VALUES or are never-served, at no extra cost, and whatever wrote the file. A refusal skips that day and is counted and printed rather than aborting the currency, so a provenance problem cannot become a coverage outage. purge_gap_chains.py now reads the delete response and re-fetches every key afterwards, reporting the count observed gone rather than the count attempted. An ambiguous probe (timeout, permission error) counts as still present. check_served_files.py is a backstop scan of every published object against the schema we publish for its dataset and the provenance enumeration. It is scheduled by check_data_contracts.sh at 06:00, which also runs check_archive_integrity.py and signup_digest.py, neither of which had ever been scheduled at all. RESOLVED 2026-10-05: the 36 objects were deleted on Henrick's instruction: 403.5 MB, 11,494,762 rows. purge_gap_chains now re-fetches every key after deleting and reported 'all confirmed gone'; an independent head_object pass over the manifest returned 36 absent, 0 present. None of the 36 had ever been downloaded (checked against all 1,354 distinct objects in file_downloads). The 530 modeled_surface files in the window were deliberately kept: they are published as model output, not passed off as real. The gap is a true hole again, matching what /methodology and the landing page have claimed throughout.

BookWriter.flush rewrote the entire day on every call. It read the day's Parquet back, converted it to Python dicts, concatenated the new deltas, rebuilt the table and wrote the whole thing out again, once per second, per currency. Cost grew with the size of the day, so each flush took longer than the last, all day, every day. The flush ran ON the event loop. While it was writing, the daemon could not read a frame from the websocket, so the time spent flushing was time spent deaf: the deltas that arrived during the write were never received at all. The gap is not a reporting artifact, it is missing data. BTC and ETH degraded in lockstep because they share the loop and flush in sequence; the second currency waits out the first. The alert on 2026-09-29 ('L2 book stale, BTC=1998s') read as a one-off stall. It was the end state of a curve that had been running for months, and it fired only when a single flush finally exceeded the 1800s threshold. Nothing watched the interval between writes, only the age of the last one, so every shorter degradation passed silently.

Impact. Time coverage of the L2 book, measured from delta timestamps inside the stored files rather than from write times: ~69 to 95% through July, degrading to 17 to 49% across 2026-09-20..29. On 2026-09-28, BTC and ETH each hold under 17% of the day. The deltas inside those windows are correct and the row counts look ordinary (11 to 13M/day), which is why the shortfall was invisible: the files are the right size and the wrong shape. No customer received this data. The L2 book is Enterprise-on-request and is not served by the files API; file_downloads contains zero book rows for all time. The loss is to our own archive. It is unrecoverable. Deribit serves no historical order book, so there is nothing to backfill from: these windows are simply gone.

Fix. flush now appends a part file and never reads what is already stored, and it runs in a worker thread instead of on the event loop. Parts are merged into the day by compact_day on day rollover, on shutdown, and before upload in sync_r2, which globs book/*.parquet non-recursively, so an uncompacted day stays invisible to the uploader rather than shipping half a day under the whole day's name. compact_day reads the merged file back and checks its row count against its inputs before deleting any part. Measured at production scale on the new path: flush is 0.16s against an empty day and 0.11s against 12M rows already on disk: flat, where the old path went from 13s to 740s over one day. A 12M-row day compacts in 2.8s, once, off the loop.