<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Money that adds up]]></title><description><![CDATA[Notes from the part of a fintech product where a wrong number costs money: payment rails, custodial wallets, double-entry ledgers and the reconciliation that ha]]></description><link>https://jeffreyjorgensen.hashnode.dev</link><image><url>https://cdn.hashnode.com/res/hashnode/image/upload/v1593680282896/kNC7E8IR4.png</url><title>Money that adds up</title><link>https://jeffreyjorgensen.hashnode.dev</link></image><generator>RSS for Node</generator><lastBuildDate>Mon, 21 Sep 2026 23:47:50 GMT</lastBuildDate><atom:link href="https://jeffreyjorgensen.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[Reconciliation doesn't break in the arithmetic]]></title><description><![CDATA[Fourteen places where the balance in your dashboard stops matching the balance at your provider and the figure in your accountant's report. For each: what the symptom looks like, and what catches it.
]]></description><link>https://jeffreyjorgensen.hashnode.dev/reconciliation-doesn-t-break-in-the-arithmetic</link><guid isPermaLink="true">https://jeffreyjorgensen.hashnode.dev/reconciliation-doesn-t-break-in-the-arithmetic</guid><category><![CDATA[fintech]]></category><category><![CDATA[database]]></category><category><![CDATA[software architecture]]></category><category><![CDATA[accounting]]></category><category><![CDATA[Testing]]></category><dc:creator><![CDATA[Jeffrey Jorgensen]]></dc:creator><pubDate>Sun, 20 Sep 2026 11:04:11 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6aafb23a740ac52935471b7d/273b9fe1-0a8b-4d46-8cee-ee0a59785a2e.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Fourteen places where the balance in your dashboard stops matching the balance at your provider and the figure in your accountant's report. For each: what the symptom looks like, and what catches it.</p>
<p>Drift almost never arrives as an outage. It arrives as a question from your accountant: “why are we four hundred and twenty short for March?” And then it turns out the system holds three numbers, each correct in its own way, and none of them can be called the balance.</p>
<p>All fourteen items below are places where two truths diverge legitimately: the provider counts one thing, the network shows another, the accountant needs a third. Without a layer that can explain the difference, the difference accumulates in silence.</p>
<blockquote>
<p>Drift in the books is not an arithmetic error. It is the absence of an answer to “where did this figure come from”.</p>
</blockquote>
<p>The list comes out of systems I built: a double-entry accounting core working across four networks (Bitcoin, Ethereum, Tron and TON), a payment rail with batch payouts, and a financial modelling system for small business, a year in daily production. Ordered from what you meet every month to what fires once a year and costs the most.</p>
<hr />
<h2>01. More left the account than the payment said</h2>
<p><em>Fee on top of the amount, or inside it</em></p>
<p><strong>Symptom.</strong> You sent a hundred, and a hundred and three left the account. Or you sent a hundred, and ninety-eight arrived. The customer calls it a bug; the system did exactly what it was told.</p>
<p>The same number means different things depending on who pays the fee. Both schemes are legitimate, but the system has to know which one it runs and say so out loud: before sending, not in reply to a complaint.</p>
<p><strong>What catches it.</strong> Three separate quantities in the payment model: amount, fee, and the resulting debit. Never display two of the three. A payout batch dry-runs first: every row returns its own fee and its own debit before any money moves.</p>
<hr />
<h2>02. The recipient was paid twice</h2>
<p><em>A retried request creates a second payment</em></p>
<p><strong>Symptom.</strong> Two requests a second apart in the logs. An operator clicked again, or the connection dropped and the client library retried on its own.</p>
<p>Retries are normal in any network exchange, not an exception. A system that creates a new operation for every request it receives is guaranteed to pay twice eventually.</p>
<p><strong>What catches it.</strong> An idempotency key from the caller, a unique index on it, and the previous result returned instead of a new one being created. A row's key inside a payout batch doesn't change between attempts, which is why re-uploading it never produces a second payment. Verified by a test that fires the same request twenty times in parallel and demands exactly one payment.</p>
<hr />
<h2>03. The balance is there and cannot be withdrawn</h2>
<p><em>A currency without its network</em></p>
<p><strong>Symptom.</strong> The dashboard says “USDT: 5,000”, the withdrawal fails: only 300 sits on the network the customer wants. They see one number and don't understand the refusal.</p>
<p>The same token lives on several networks, and that's not one asset but several. Adding them into a single number is like adding cash in your pocket to store credit at a shop that's shut: you get a figure, and none of it is spendable where the customer wants to spend it.</p>
<p><strong>What catches it.</strong> The pair “currency plus network” becomes the key everywhere: in the account, in the posting, in the API response, in the exported row. Amount, currency and network travel together and are never separated. Addresses are validated against the format of that specific network before anything is queued.</p>
<hr />
<h2>04. The balance went negative despite the check</h2>
<p><em>Accounting without holds</em></p>
<p><strong>Symptom.</strong> The balance is below zero. There's a “sufficient funds” check in the code and it was running.</p>
<p>Time passes between reading the balance and debiting it. Two concurrent requests read the same value, both see enough, both debit. The check existed; isolation did not, and a transaction alone doesn't supply it. Under PostgreSQL's default <code>READ COMMITTED</code>, both requests read the same available balance, both insert their own hold, neither write conflicts with the other, and both commit.</p>
<p><strong>What catches it.</strong> Funds are reserved as their own object, a hold, and the available balance is that balance minus active holds. Check and reserve happen inside one transaction <em>and</em> serialise against each other: the account row is taken with <code>SELECT … FOR UPDATE</code>, or the debit is a single conditional <code>UPDATE … WHERE available &gt;= amount</code> whose row count decides the outcome, or the transaction runs at <code>SERIALIZABLE</code> and retries on conflict. In a payout batch the remaining balance is a running total: rows that no longer fit are flagged in advance rather than failing at dispatch.</p>
<hr />
<h2>05. Daily reports don't add up to the monthly one</h2>
<p><em>The day boundary and time zones</em></p>
<p><strong>Symptom.</strong> The gap is exactly a handful of transactions, and all of them happened near midnight.</p>
<p>A transaction happens at an instant; a report is built on calendar boundaries. When different parts of the system understand “a day” differently — the server in one zone, the database in another, the accountant in a third — some transactions land in both reports or neither.</p>
<p><strong>What catches it.</strong> Every instant is stored in one zone with the zone stated explicitly, the reporting period is an interval rather than a date, and the reporting zone is written into the report itself. Test: a transaction exactly on the boundary must land in precisely one report.</p>
<hr />
<h2>06. Turnover doubled out of nowhere</h2>
<p><em>A refund recorded as a new operation</em></p>
<p><strong>Symptom.</strong> The turnover metric doubled with no growth in sales. Refunds were counted as fresh inflows.</p>
<p>A refund can be recorded two ways: as a separate operation in the opposite direction, or as a reversal of the original. The first is correct for money and wrong for turnover: one deal now produces two movements, and any metric that counts movements doubles.</p>
<p><strong>What catches it.</strong> A refund references its original operation explicitly and carries its own type. A partial refund holds its own amount and never edits the original record. Turnover is computed over operations of a given type, not over every movement. A database constraint: refunds against an operation cannot exceed its amount.</p>
<hr />
<h2>07. One transaction, three different figures</h2>
<p><em>A rate as of exactly when</em></p>
<p><strong>Symptom.</strong> Last quarter's report built today differs from the same report built a month ago. All three versions are “correct”.</p>
<p>The rate moves between a payment being created, sent and confirmed. If the system takes the rate at the moment the report is built, past transactions change value retroactively, and the books stop being reproducible.</p>
<p><strong>What catches it.</strong> The rate is fixed with the posting and stored inside it: not a reference to a table, but the value itself and the instant it was taken. A report for a past period, built today and a year from now, must produce identical numbers. The same requirement is what makes an audit possible.</p>
<hr />
<h2>08. Yesterday's export names the source differently</h2>
<p><em>A reference table edited retroactively</em></p>
<p><strong>Symptom.</strong> Two exports covering the same period call the same partner by different names. Or a tariff changed and past fees recalculated themselves.</p>
<p>The posting references a table, the table changes, history changes with it. The quietest problem on this list: nothing breaks, the past simply stops holding still.</p>
<p><strong>What catches it.</strong> Everything that affects a calculation is captured in the posting at the moment it's created: the tariff rate, the version of the rules, the name of the source. The label can change as often as you like; it's for humans. The key that totals reconcile against never changes, and those are two different fields. This is the only entry on this page the checker doesn't implement: seeing that a row was rewritten needs both the old and the new state of the table, and given both the detector is a diff of two files. This one is prevented by how the data is stored, not caught afterwards.</p>
<hr />
<h2>09. A confirmed deposit disappeared</h2>
<p><em>Finality, not “confirmations”</em></p>
<p><strong>Symptom.</strong> A deposit was credited and an hour later it isn't on the network. The customer's balance already grew and the money is already spent. Or the reverse: the transaction is visible and never makes it into a block.</p>
<p>Four networks run three different models of finality and four different thresholds, and no single number covers them. In Bitcoin finality is probabilistic; at zero confirmations the transaction is also replaceable. In Ethereum finality is crypto-economic: before <code>finalized</code> a reorg across a few slots is possible. In Tron and TON finality rests on validator agreement, and waiting ten more blocks doesn't fix a rollback past the irreversibility boundary.</p>
<p><strong>What catches it.</strong> The threshold comes from the network's model, not from a round number: in Bitcoin a count of confirmations, in Ethereum the <code>finalized</code> tag, in Tron block irreversibility, in TON the masterchain signature. Below the threshold the deposit sits on a separate pending account and shows to the customer as waiting. A credit is reversed with an opposing posting, never by deleting.</p>
<hr />
<h2>10. One transaction is not one operation</h2>
<p><em>Batches, sweeps and asynchronous transfers</em></p>
<p><strong>Symptom.</strong> The chain watcher saw one transaction and created one posting; there were two hundred recipients inside it. Or the reverse: one payout spawned a chain of messages and part of it bounced back.</p>
<p>The one-transaction-one-operation assumption breaks in three places at once. A batch with many outputs is one transaction and many payments. Moving funds to a central wallet is a network movement with no customer operation behind it. And on TON a token transfer is a chain of messages, any of which can bounce midway.</p>
<p><strong>What catches it.</strong> The relationship between operations and transactions is many-to-many, not one-to-one, and that's in the schema from the start. Internal movements carry their own type and never pose as customer activity. For asynchronous transfers, success means a confirmed credit on the recipient's side, not a successful send. Until then the operation stays open.</p>
<hr />
<h2>11. Reconciliation fails though every entry is right</h2>
<p><em>Internal movements missing from the equation</em></p>
<p><strong>Symptom.</strong> “Sum of customer balances equals the on-chain holding” never holds, and the gap is different every day.</p>
<p>Not everything that moves money is in the equation. Transfers between customers inside the system never touch the network. But sweeps to the central wallet do, and so does topping addresses up with gas, and so do the fees themselves, and none of those have a customer posting.</p>
<p><strong>What catches it.</strong> Postings carry a type, and reconciliation is built on the full equation: the change in on-chain holdings equals external inflows minus external outflows, minus fees, minus gas movements, with internal transfers excluded entirely, though they must sum to zero across the system. Each term is computed separately, so a discrepancy points at where to look.</p>
<hr />
<h2>12. The cost of a payout changed after sending</h2>
<p><em>The actual fee is not the estimated one</em></p>
<p><strong>Symptom.</strong> The cost of a payout in the report doesn't match what was shown at dispatch. Sometimes by a lot.</p>
<p>The fee at send time is an estimate, not a fact. A transaction can stall and be accelerated, and then the cost of an already-sent payout changes retroactively. On networks with dynamic gas pricing, a gap between expected and actual is ordinary.</p>
<p><strong>What catches it.</strong> Estimated and actual fees are two separate fields, and the second is filled from the network after confirmation. The difference isn't hidden: it becomes its own posting, and that posting is what shows the real cost of running payouts. Accelerating a transaction is an event with its own record.</p>
<hr />
<h2>13. The balance exceeds what can be withdrawn</h2>
<p><em>Dust, minimum reserves and freezes</em></p>
<p><strong>Symptom.</strong> Total system balance is larger than what can actually be sent. The difference matches no single transaction.</p>
<p>Three legitimate causes. Remainders smaller than the network fee cost more to move than they're worth. Some networks require a minimum reserve on an account. And a stablecoin issuer can freeze an address. The funds are there and can't be moved, and that's neither your mistake nor your choice.</p>
<p><strong>What catches it.</strong> Withdrawable balance is a separate quantity, not a synonym for balance. Both appear side by side in the report, and every part of the gap has a name: dust, reserve, frozen. A freeze is recorded as a restriction on the account, not as funds vanishing. A customer needs an explanation, not a quietly smaller number.</p>
<hr />
<h2>14. The payout went out and cost nothing</h2>
<p><em>A fee paid in a different asset</em></p>
<p><strong>Symptom.</strong> The report shows a payout with zero cost. Or an asset was debited that has no business being in this operation.</p>
<p>A fee doesn't have to be paid in the asset being transferred. It can come out of the network's base coin, be covered by a resource rented in advance, or be paid by a third party entirely. Books where an operation has exactly one currency can't record that cost and lose it silently.</p>
<p><strong>What catches it.</strong> The fee is its own posting with its own currency and network, linked to the operation but not part of its amount. A resource bought in advance is accounted for as inventory and drawn down as it's consumed. The total cost of a payout is the sum of postings across all currencies, converted at the rates already fixed.</p>
<hr />
<h2>Thirteen of these are checkable</h2>
<p>Everything above says how to build a system that doesn't drift. <a href="https://github.com/jeffreyjorgensen/ledger-reconcile">ledger-reconcile</a> answers the other question: whether yours already has. It takes what your ledger says happened and what your provider recorded, and reports where the two stop agreeing. It doesn't implement the fixes above: it finds out which of them you're missing.</p>
<p>Every finding names the entry on this page that explains it, carries the records that produced it, and writes out the arithmetic that failed. It also reports what it did <em>not</em> check: a run lists every check it couldn't perform and which input was missing, because a tool that quietly performed four checks out of twelve and printed “no findings” would be worse than no tool.</p>
<p>Thirteen of the fourteen are implemented; the fourteenth is number 08, and the reason is in the README rather than left for you to find. One dependency, a decimal type for money, which is the fifteenth item above, and the reason it isn't on the list.</p>
<hr />
<p><em>First published at <a href="https://jeffreyjorgensen.dev/teardown?from=hashnode">jeffreyjorgensen.dev/teardown</a>. Two more in the same series: <a href="https://jeffreyjorgensen.dev/security?from=hashnode">The check that never failed</a> and <a href="https://jeffreyjorgensen.dev/falsehoods?from=hashnode">Falsehoods engineers believe about moving money</a>.</em></p>
]]></content:encoded></item><item><title><![CDATA[The check that never failed]]></title><description><![CDATA[Fourteen measures that were present, had been reviewed, and were believed. None of them was working. Each entry says what it looked like from outside, and what it took to make it prove itself.
A safeg]]></description><link>https://jeffreyjorgensen.hashnode.dev/the-check-that-never-failed</link><guid isPermaLink="true">https://jeffreyjorgensen.hashnode.dev/the-check-that-never-failed</guid><category><![CDATA[Security]]></category><category><![CDATA[Devops]]></category><category><![CDATA[Testing]]></category><category><![CDATA[software architecture]]></category><category><![CDATA[fintech]]></category><dc:creator><![CDATA[Jeffrey Jorgensen]]></dc:creator><pubDate>Sun, 20 Sep 2026 10:53:28 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6aafb23a740ac52935471b7d/dde2522c-c3bb-4946-a9b5-cf339b1de0ca.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Fourteen measures that were present, had been reviewed, and were believed. None of them was working. Each entry says what it looked like from outside, and what it took to make it prove itself.</p>
<p>A safeguard has two states that are easy to confuse: present, and working. Almost nothing below was found by an incident. Each was found by asking what would have to be true for the measure to be working, and then attempting it.</p>
<p>None of them were found by reading code either. Reading tells you the control is there. It can't tell you the control is reached, that its refusal path has ever executed, or that the permission it depends on was actually granted.</p>
<blockquote>
<p>A check that has never shown a failure has not been checked.</p>
</blockquote>
<p>Every entry came out of a system I had to prove rather than one I had read: an identity service issuing signed tokens, a double-entry accounting core across four networks, and the infrastructure under both. Ordered from what you meet on every deploy to what costs the most when it fires.</p>
<hr />
<h2>01. The service that started in the wrong mode</h2>
<p><em>An absent value is not a typo</em></p>
<p><strong>What you see.</strong> Production runs as development, and nothing in the logs or the health checks looks wrong.</p>
<p>A configuration field typed as a fixed set of values catches a misspelling. It catches an empty string. It doesn't catch absence, because absence takes the default, and the permissive mode is a popular default.</p>
<p>The damage isn't the mode itself, it's what hangs behind it: a production system that quietly enables its development affordances is indistinguishable, in its logs and in its health checks, from one that started correctly. There's no moment at which anything looks wrong.</p>
<p><strong>What proves it.</strong> The environment name has no default, and the service refuses to start without it. The test asserts the refusal, not the parsing: the parsing was never the part that failed.</p>
<hr />
<h2>02. The stored data that a config change made unreadable</h2>
<p><em>The ciphertext says how to read it; the settings don't</em></p>
<p><strong>What you see.</strong> After a configuration change, data written earlier stops decrypting, and the error blames the key.</p>
<p>If the decryption scheme is selected by configuration, then changing configuration retroactively reinterprets data written under the old one.</p>
<p>The failure surfaces as “wrong key”. That's the expensive part: whoever is debugging goes looking for a key problem that doesn't exist, while the data is intact and merely being read by the wrong rules. Every minute of that hunt is spent in the wrong repository.</p>
<p><strong>What proves it.</strong> The scheme is named by a prefix on the ciphertext itself, and no prefix means legacy. Configuration cannot reinterpret history it did not write.</p>
<hr />
<h2>03. The risk that was described and stayed open</h2>
<p><em>A name is not a fix</em></p>
<p><strong>What you see.</strong> The defect is documented, everybody knows about it, and it's still there.</p>
<p>A test can carry a risk in its name and assert nothing about it. A comment can state a hole precisely and leave it exactly where it is.</p>
<p>This is the failure mode of teams that document well. A described gap feels handled in a way an undescribed one doesn't, and it acquires a strange kind of immunity: everybody has read it, so nobody is surprised by it, so nobody is alarmed. It can sit in plain sight for weeks with no one hiding anything.</p>
<p><strong>What proves it.</strong> An open risk gets an assertion that fails until it closes, or a line in a list the document refuses to omit. An empty list reads as “covered”, which is the one thing it was never safe to assume.</p>
<hr />
<h2>04. The mechanism that was built and never ran</h2>
<p><em>Written, permitted, unreachable</em></p>
<p><strong>What you see.</strong> The function exists, the permission for it was granted by its own line in the access policy, and it's called from nowhere.</p>
<p>A separate, deliberate line in an access policy is evidence that at least two people reasoned about this mechanism. It still never ran.</p>
<p>Dead code is ordinarily harmless. This kind isn't, because everyone downstream has already recorded the operation it belongs to as done. The mechanism's existence is doing the work its execution was supposed to do.</p>
<p><strong>What proves it.</strong> A check that every public mechanism has a caller, run in CI where a new one can't be added without it.</p>
<hr />
<h2>05. The green check that was always going to be green</h2>
<p><em>An indicator is not a diagnosis</em></p>
<p><strong>What you see.</strong> The check is green, and it would have been green either way.</p>
<p>A source scan that looks for variable names finds the names it knows. It doesn't find a value arriving under a different name, a foreign response body logged whole in one call, or a variable somebody renamed last week.</p>
<p>Its green means “not found this way”. That's a much smaller claim than the one people read into it, and the gap between the two is where the rule quietly stops being enforced while continuing to be reported.</p>
<p><strong>What proves it.</strong> Publish the limitation beside the check, in the same table as the claim it supports. An indicator presented as a diagnosis retires the question it was meant to keep open.</p>
<hr />
<h2>06. The control that stopped applying and said nothing</h2>
<p><em>A silent failure is a security defect, not an inconvenience</em></p>
<p><strong>What you see.</strong> The measure no longer applies, and nobody reported it.</p>
<p>A mandatory input that falls back to a default when it's missing. An exception swallowed on a path that only executes in production. A filter that passes quietly when its pattern fails to match.</p>
<p>Each is a control that stopped working while continuing to look present, and each will be discovered by exactly the event it existed to prevent. Silence isn't neutral here: it's the report that the control is fine.</p>
<p><strong>What proves it.</strong> The absence of a required input is a refusal, not a default, and the refusal has its own test, because a refusal nobody has triggered is a refusal nobody has watched work.</p>
<hr />
<h2>07. Healthy, ready, and unable to do the job</h2>
<p><em>Readiness checked the dependency, not the capability</em></p>
<p><strong>What you see.</strong> Both probes green, every authenticated request failing.</p>
<p>A readiness probe that asks whether the key service answers is checking the wrong thing. Between “the vault answers” and “we can sign” sit a sealed vault (running and reachable, but not yet handed its keys after a restart), an expired role credential, and a value encrypted under a scheme that no longer exists. All three leave the dependency perfectly reachable and the capability gone.</p>
<p><strong>What proves it.</strong> Readiness performs the capability and fails when it can't. Pair it with the opposite rule, which matters more: the same sealed state must break issuing and must not break verifying, or a routine operational pause turns into a full stop for everything downstream.</p>
<hr />
<h2>08. The request that chose how much it would cost</h2>
<p><em>The sender sets the price unless you do</em></p>
<p><strong>What you see.</strong> One request stalls the whole process, health endpoint included.</p>
<p>A request body with no ceiling reaches a parser, and the parser's cost is chosen by whoever sent the bytes.</p>
<p>Measured on my own code: 640 KB of a structured binary format spent 29.5 seconds of CPU inside a single-threaded event loop. Everything else waited, including the health endpoint, which is how a parsing problem presents as an outage.</p>
<p><strong>What proves it.</strong> A ceiling on body size, decided as a cost question rather than set as a configuration value. Be exact about what it isn't: it isn't a rate limit, and it doesn't mean the bytes failed to arrive.</p>
<hr />
<h2>09. The service that answered a question it shouldn't have</h2>
<p><em>Answer with a fact, not a decision</em></p>
<p><strong>What you see.</strong> One service decides what another service is allowed to do.</p>
<p>“The session exists” and “this is allowed” are different sentences, and the identity layer is only entitled to the first.</p>
<p>When it answers the second, its notion of sufficiency is compiled into every caller. The day one caller needs a stricter bar (a larger transfer, a new jurisdiction, a customer who asked for it), the change has to be made in the wrong codebase, by a team that doesn't hold the risk and can't price it.</p>
<p><strong>What proves it.</strong> Return facts (which methods proved the identity, when, at what strength) and let the caller decide. It reads as less helpful. It is what keeps the decision next to the money.</p>
<hr />
<h2>10. The strong factor that a weak one could remove</h2>
<p><em>A construction is as strong as the cheapest way to dismantle it</em></p>
<p><strong>What you see.</strong> The account is protected by a strong second factor, and that factor can be removed by presenting a weaker one.</p>
<p>Adding a strong second factor raises the bar exactly until the recovery path lowers it again.</p>
<p>If a strong factor can be removed by presenting a weaker one, then the account is protected by the weaker one, whatever the enrolment screen implies. The arithmetic isn't additive. It's a minimum, taken over every path that can remove a factor, and recovery paths are usually designed by whoever is optimising for support volume.</p>
<p><strong>What proves it.</strong> Removing a factor requires that same factor, not merely a strength level that something weaker also satisfies. The distinction is between <em>which method proved it</em> and <em>how strong the proof was</em>: two claims that collapse into one at the first convenient moment.</p>
<hr />
<h2>11. The rotated key that kept working</h2>
<p><em>Rotation is a change and a withdrawal</em></p>
<p><strong>What you see.</strong> The rotation is done, both sides consider it complete, and the old key is still accepted.</p>
<p>Rotation is two operations, and the second is the one that gets lost.</p>
<p>Taking the old key out of circulation has to happen everywhere verification reads, and there's usually more than one such place: a published key set for outside callers, a direct lookup for internal routes. Filter one and the other carries on accepting, which is worse than not rotating at all, because now everybody believes it's done and the compromised material is no longer being watched.</p>
<p>The same shape appears one layer down. A key service reports a rotation and the rotation happened; what didn't happen was raising the minimum version accepted for decryption, so the retired key goes on unwrapping old material. Both sides read the success of an operation as the arrival of a property.</p>
<p><strong>What proves it.</strong> Withdrawal lives in the store that every verification reads, not inside each verification. The command exits non-zero until both halves are done. A command that exits zero at the halfway point announces “done” in precisely the place where it's half done. And the test rotates, then presents the old key to every path there is.</p>
<hr />
<h2>12. The audit log its own subject could edit</h2>
<p><em>A witness table is defined by permissions, not by intent</em></p>
<p><strong>What you see.</strong> The log is complete, and the role whose actions it records can rewrite it.</p>
<p>A table becomes evidence when the role whose actions it records can't change it. A comment in the model reading “audit log — do not modify” is an intention, and intentions are not controls.</p>
<p>The argument extends to shape: a log whose schema one person can change alone is not a log, for the same reason that a log one person can empty is not one.</p>
<p><strong>What proves it.</strong> A separate owning role; insert and select for the recorded role and nothing else. And somebody has actually attempted update, delete, truncate and alter from that role and watched each one refuse, because this property lives in the live database and nowhere else. On a test database the role is different and the grants don't exist at all, so the suite can show that the log records and can never show that the log resists. The half it covers is not the interesting half.</p>
<hr />
<h2>13. The probe that reported the opposite of the truth</h2>
<p><em>A zero exit code is not evidence</em></p>
<p><strong>What you see.</strong> The permission check passed. The permission was never granted.</p>
<p>Granting yourself a privilege you don't hold can succeed without error and without effect.</p>
<p>A probe that reads only the return code reports success, and reports it in the one place where being wrong costs most, because this is the check that exists to catch exactly this. The probe isn't broken. It's answering a question about the command, not about the world.</p>
<p><strong>What proves it.</strong> Probe by consequence, never by exit status. Attempt the forbidden operation and require it to fail.</p>
<hr />
<h2>14. The audit record deleted along with its subject</h2>
<p><em>Cascade is a convenience that erases evidence</em></p>
<p><strong>What you see.</strong> The user was deleted, and with them the record of what was done to them.</p>
<p>A cascading foreign key is the obvious default, and it quietly makes the evidence weaker than the thing it witnesses: a row nobody is permitted to delete directly disappears the moment its subject does.</p>
<p>Nobody decides this. It arrives with the schema, it survives review because it's the normal choice everywhere else, and it's discovered on the day somebody asks what was done to an account that no longer exists.</p>
<p><strong>What proves it.</strong> The reference nulls instead of cascading, verified by behaviour against a live database rather than by reading the schema, because the schema is what looked correct in the first place.</p>
<hr />
<h2>Every one of them reported success</h2>
<p>Every one is a measure that reported success while not applying — because nothing ever asked it to refuse, and a refusal nobody has triggered is a refusal nobody has watched work.</p>
<blockquote>
<p>A control is worth what its failure path is worth, and that path is the one nobody runs.</p>
</blockquote>
<p>Two of the catches above are source scans, and both are labelled as indicators rather than proof. Everything else was found the same way: presenting a withdrawn key to every verification path there is, running the forbidden statement as the role that must not be able to run it, sending the request that costs too much. The list is kept in the open because an empty line in a document like this reads as “covered”.</p>
<hr />
<p><em>First published at <a href="https://jeffreyjorgensen.dev/security?from=hashnode">jeffreyjorgensen.dev/security</a>. Two more in the same series: <a href="https://jeffreyjorgensen.dev/teardown?from=hashnode">Reconciliation doesn't break in the arithmetic</a> and <a href="https://jeffreyjorgensen.dev/falsehoods?from=hashnode">Falsehoods engineers believe about moving money</a>.</em></p>
]]></content:encoded></item><item><title><![CDATA[Falsehoods engineers believe about moving money]]></title><description><![CDATA[A penny I sent myself switched off the balance check. The coins went out to the network; nothing was debited.
It wasn't a typo and it wasn't a missing check. It was a sentence I believed: an idempoten]]></description><link>https://jeffreyjorgensen.hashnode.dev/falsehoods-engineers-believe-about-moving-money</link><guid isPermaLink="true">https://jeffreyjorgensen.hashnode.dev/falsehoods-engineers-believe-about-moving-money</guid><category><![CDATA[golang]]></category><category><![CDATA[fintech]]></category><category><![CDATA[Security]]></category><category><![CDATA[software architecture]]></category><category><![CDATA[Blockchain]]></category><dc:creator><![CDATA[Jeffrey Jorgensen]]></dc:creator><pubDate>Sun, 20 Sep 2026 10:39:26 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6aafb23a740ac52935471b7d/bff9b28a-11cf-46d3-91c0-36c7173f2edf.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>A penny I sent myself switched off the balance check. The coins went out to the network; nothing was debited.</p>
<p>It wasn't a typo and it wasn't a missing check. It was a sentence I believed: <em>an idempotency key makes a retry safe</em>. It reads like an axiom. It's false.</p>
<p>Ten sentences below, that one first. All of them about moving money: holds, deposits, fees, reserves, asset precision. I believed every one, and every one cost real money. Under each there is a commit and a test you can open. Some are covered in other people's CVEs and audits, and I'll point at them. For some I found no public write-up at all, and I did look.</p>
<hr />
<h2>01. An idempotency key makes a retry safe</h2>
<p>The retry, yes. Which checks that retry is allowed to skip, the key says nothing about. Whoever wrote the branch decides.</p>
<p>Ours skipped the balance check. The reasoning was simple and looked sound: if this is a replay, the request already passed every check once, so why run them again. That is why the branch sat first, ahead of the type, account, amount and funds comparisons.</p>
<p>And we built the hold's key by concatenation: <code>external_ref + ":hold"</code>. So the key wasn't invented by the server. It was invented by whoever turned up.</p>
<p>The rest is arithmetic. Send yourself a penny with <code>external_ref</code> set to <code>X:hold</code>. Then create a withdrawal with <code>external_ref</code> = <code>X</code>. The server builds the hold's key as <code>X</code> + <code>:hold</code>: exactly the one your penny is already filed under. The hold finds it, says "ah, a replay, all done", and reserves nothing. The coins go to the network. The account is not debited.</p>
<p>It worked end to end on TRC20 and TON. Each network has its own send path; we call them rails. On the Bitcoin and Ethereum rails the balance was checked a second time, down there, so it wouldn't have gone through. We learned that afterwards, taking it apart, not because anyone planned it.</p>
<p>Idempotency keys have been written about for years, and the happy path is well covered. The unhappy one has been covered recently too: <a href="https://blog.dochia.dev/blog/idempotency/">"Idempotency is easy until the second request is different"</a> has the canonical-command comparison, the ordering relative to authorisation, and mutable state such as balance. What I found nowhere, that post included, is <strong>a derived key the server synthesises from a field the client sent</strong>. Everyone writes about a key's scope. Nobody writes about what it was glued together from.</p>
<p><strong>What catches it.</strong> A replay has to prove it is the same operation, not merely the same string: compare type, account and amount before you hand back a stored response. And build the key for a derived step on the server, out of the request itself, instead of pasting a suffix onto something the client supplied. Ours is <code>TestHold_PlantedTransactionUnderSameRef_IsRefused</code>.</p>
<hr />
<h2>02. A deposit monitor that's running loses nothing</h2>
<p>The other way round. A broken one loses nothing; a healthy one loses everything.</p>
<p>The monitor walked the chain and moved its checkpoint after each block it scanned. A transaction that hadn't reached the confirmation depth it skipped: a line in the log saying "waiting for confirmations", and on to the next block. Reasonable enough: we'll pick it up on the following pass.</p>
<p>There was no following pass. By then the block was marked processed, and nothing will ever read it again.</p>
<p>Now the inversion. It scanned right up to the tip, and the interval on Ethereum is fifteen seconds. So it saw a fresh transaction a couple of blocks after it appeared, always before the confirmations had accrued. Every deposit that arrived while the monitor was running was lost. And if the monitor had been down for an hour and was catching up, it saw blocks that already had their confirmations and credited all of them correctly.</p>
<p>Crypto has no name for this. Streaming does: it's committing the offset before processing, and the Kafka documentation warns about it in as many words. The only difference is that streaming has no notion of confirmation depth, which is why the "healthy loses, lagging doesn't" inversion can't happen there.</p>
<p><strong>What catches it.</strong> Scan to <code>tip − MinConfirmations</code>, not to the tip. After that the "waiting for confirmations" branch is unreachable in normal operation. The confirmation count itself is an assumption rather than an invariant: on proof-of-stake the right boundary is consensus finality, not a block counter. Our test asserts the cursor's position, that it never went past <code>tip − MinConfirmations</code>, and that is weaker than I'd like: the branch is still there in the code.</p>
<hr />
<h2>03. Checking an amount is cheap</h2>
<p>Parsing <code>{"amount":"1e1000000000"}</code> genuinely is. Mantissa one, exponent a billion, twelve bytes on the wire. The number is not expanded at parse time. It is expanded at the first arithmetic operation.</p>
<p>The first arithmetic operation is our check. "Amount is positive." "Amount is within the limit." A comparison against zero looks like the cheapest operation in the world, and it aligns both operands to a common exponent.</p>
<p>I measured it rather than guess at the order of magnitude. Go 1.26, arm64, <code>shopspring/decimal</code>; the clock covers exactly one <code>GreaterThan(Zero)</code> after parsing:</p>
<pre><code>1000.50       0.0 ms    +0 MB
1e100000      0.6 ms    +0 MB
1e1000000      20 ms    +3 MB
1e5000000     251 ms   +21 MB
</code></pre>
<p>Parsing is free in all four. The comparison pays.</p>
<p>A separate piece of nastiness: the bound was present in six methods, but the deposit path with a non-zero fee didn't go through a guarded one. So the check switched itself on and off according to a pricing setting.</p>
<p>In Java the class is known and closed: the Johnzon CVE is literally about <code>1e20000000</code>, and the fix was a scale limit on BigDecimal. Meanwhile the industry default, Jackson's <code>maxNumberLength=1000</code>, bounds the <strong>literal's length</strong>: it catches a long mantissa and lets a short exponent through. Two projects, two different bounds, and neither is about the exponent. <code>shopspring/decimal</code> has zero advisories in OSV at all.</p>
<p><strong>What catches it.</strong> Check the exponent and the number of significant digits. <code>Exponent()</code> and <code>NumDigits()</code> don't expand the mantissa, so the rejection costs nothing. A length bound won't catch this shape: our twelve bytes clear any such limit. You still want the length bound, because a long mantissa is exactly what it does catch, and it belongs first, before the number is built. And remember that the rejection has to be cheap: a validator that refuses a number at the cost of a gigabyte has closed nothing. In Go that matters more than elsewhere: running out of memory is fatal, it kills the process rather than the request, and no middleware will intercept it.</p>
<hr />
<h2>04. An incoming transfer to a user's address is a deposit</h2>
<p>Usually. Except when the sender is you.</p>
<p>A sweep collects money from deposit addresses into the hot wallet, and on EVM it is impossible without gas sitting on the address itself. So the platform sends ETH to the customer's deposit address in order to take tokens off it afterwards. By address alone that transfer is indistinguishable from a real deposit: in both cases money moves from the hot wallet to a user's address.</p>
<p>The monitor saw it as an ordinary incoming transfer and credited it. The top-up default is 0.003 ETH, and that much accrued out of nowhere every cycle. Liabilities grew while the on-chain funds belonged to the platform throughout, and the reserve check drifted along with them.</p>
<p>Filtering by address cannot work here, in principle. The only source of truth is your own registry of your own transaction hashes, which the sweeper writes and the monitor reads.</p>
<p>Vitalik has written about exchanges shuffling collateral between each other to perform solvency. That is asset inflation, and it is deliberate. This is liability inflation, and it is accidental: caused by your own housekeeping traffic.</p>
<p><strong>What catches it.</strong> A registry of internal transfers, and a monitor that checks against it. And while you're there, look at your units: the same commit fixed a muddle where the balance arrived in wei, the gas cost was subtracted in ETH, and the result was multiplied by 10¹⁸ a second time. The sweeper tried to send a quintillion times the balance.</p>
<hr />
<h2>05. A send error means nothing went out</h2>
<p>For us, <strong>any</strong> send error released the hold except a hot-wallet shortage. Unconditionally, on all four rails: it didn't send, so put the money back.</p>
<p>Alongside it, on the Bitcoin rail, lived a list called <code>nonRetryablePatterns</code>, commented "fundamental issues with the transaction", and among its entries sat <code>txn-mempool-conflict</code>. That list decided whether to stop retrying.</p>
<p>Both constructions rest on the same premise: the send returned an error, so nothing went out.</p>
<p>That doesn't follow from an error. Classify by stage, not by text. Building, encoding and signing all happen before the network is touched, so their failure proves the transaction does not exist. Anything that fails after the network call proves only that you don't know. There are three outcomes, not two: it went, it definitely didn't go, and unknown.</p>
<p>Card processors solved this with a header: at Stripe, the absence of <code>Stripe-Should-Retry</code> in a response means "we cannot determine whether this is safe to retry." Blockchains have no such header, so everyone matches strings, even though the nodes themselves classify by class rather than by wording. In go-ethereum, <code>ErrAlreadyKnown</code> is documented as "the transactions is already contained within the pool"; Bitcoin Core has <code>TX_CONFLICT</code>, commented "Tx already in mempool or conflicts with a tx in the chain".</p>
<p>And there you can see why strings are a poor source of truth. In Bitcoin Core, <code>txn-mempool-conflict</code> means a conflict with a transaction <strong>in</strong> the mempool: a double spend of the same input. It does not prove that yours will go through. We read it as "all good, it's already in there", which was a guess from a string like every other guess from a string.</p>
<p><strong>What catches it.</strong> Release a hold only on proven absence. In every other case keep it and call a human. A stuck withdrawal is resolved by hand; a second payout in non-reversible coins is resolved by nothing.</p>
<hr />
<h2>06. A column of the right shape is a key</h2>
<p>A batch of payouts arrives as a file, and that file usually has its own column of idempotency keys. There may be no header row at all, and if there is one it's in the customer's language, so we guess the column by its contents.</p>
<p>And a column is a key only if its values are distinct. Without that check the period column would have found its way into the idempotency keys, "2026-08" in every row, and the second line would have been refused as a duplicate: someone doesn't get paid because of our guess. That didn't happen: role detection and the uniqueness condition landed in the same commit. The other direction did happen.</p>
<p>Someone pasted an export with their own keys, the detector didn't recognise them, the keys quietly vanished, and the batch went out under keys we invented. A second upload of the same file produced different invented ones: that is, a second payment. Precisely where the key was supposed to stop it.</p>
<p>Column auto-detection looks like a convenience. What it actually does is decide whether the product's principal safeguard works at all, and it can fail in both directions.</p>
<p><strong>What catches it.</strong> A uniqueness check on the values: without it, a "key" is just the first column of the right shape. Ours looks at the first twenty non-empty rows, and that's a compromise rather than a solution: a file whose repeats begin on the twenty-first row will pass. Plus a rule about guessing: if you recognised nothing, return the original order rather than guessing half of it. Half-guessed is worse than not guessed, because the operator reads it as "the system worked it out" and stops checking.</p>
<hr />
<h2>07. Address case is a property of the network</h2>
<p>It's a property of the format.</p>
<p>bech32 may be written entirely in capitals: BIP-173 carries a test vector for it, and a decoder is required to accept it on equal terms with the lowercase form. The only thing forbidden there is mixed case. We compared against the sanctions list case-sensitively, with the rules split by network. The same address, submitted in capitals, didn't match. The check answered "clear".</p>
<p>Base58, meanwhile, is genuinely case-sensitive; there the case carries information. So the rule cannot be one per network: a single chain hosts formats with opposite requirements, and normalisation has to follow the format, not the rail.</p>
<p>NVD has zero entries for <code>bech32</code>. Zero for <code>EIP-55</code>. The class has no name, even though both specifications describe in plain words the ambiguity it grows out of: ERC-55 goes as far as listing compatibility with mixed-case-accepting parsers as an advantage.</p>
<p><strong>What catches it.</strong> Normalisation by format, sitting next to normalisation by network, in the same comparator. One does not replace the other: the per-network rule still does real work where case genuinely is a property of the rail. And a test that an address in capitals is still matched by the same sanctions list: <code>TestScreen_Bech32UppercaseStillMatches</code>.</p>
<hr />
<h2>08. A signing policy limits what leaves</h2>
<p>It limits the transfer amount. What leaves the account, it doesn't limit at all.</p>
<p>The fee is the hot wallet's second till, and on EVM the caller names it: <code>gas × gas_price</code>, both fields his. A one-penny transfer with a fee equal to the entire remaining balance satisfied the rules: the amount ceiling was respected, and the wallet was empty.</p>
<p>On Bitcoin it's worse, and that part isn't our omission. There the fee equals inputs minus outputs, and for legacy inputs the transaction being signed does not carry the input amounts at all. An isolated signer, which by construction never reaches the chain, physically cannot compute it. Taking it as declared by the caller is pointless: the caller is precisely the party being distrusted.</p>
<p>This is the motivation for BIP-143, stated in the document in plain words: for an offline device, not knowing the input amount makes the fee impossible to compute. Except it says this about a cold wallet, an offline signing device. I found no formulation anywhere applying it to a custodial signing policy, even though the position is identical: a signer who must not trust its caller.</p>
<p><strong>What catches it.</strong> Before segwit, a bound on the untrusted side, expressed as the ratio of fee to withdrawal amount. After it, the fee becomes verifiable at the signer itself, and that is where it must be verified.</p>
<p>Then two conditions without which a ceiling isn't a ceiling. First: a per-transaction limit bounds nothing while requests can arrive concurrently; the exposure equals the ceiling multiplied by the number of parallel signatures. Second: the output sum is computed with an overflow check. Four outputs of 2⁶² each come to exactly zero; a fifth of a hundred satoshi makes the total small and positive, and the policy sees a hundred satoshi where the transaction pays eighteen quintillion.</p>
<hr />
<h2>09. A solvency circuit breaker protects money</h2>
<p>It freezes it.</p>
<p>The reserve check compares liabilities against what sits on-chain at our addresses. The address enumerator filtered on <code>is_active = TRUE</code>, which seemed reasonable: we only ever deactivate addresses that were never used.</p>
<p>The argument was true of the application's invariant and false of the database. The code does deactivate only unused addresses; it checks. But migrations write rows that no code path can produce, and among them are rows with money on them. On the staging environment that dropped 1.522 ETH against liabilities of 1.5246: the check saw zero across the addresses it got from the database, i.e. a shortfall roughly the size of the entire liability. Under <code>RECONCILIATION_ENFORCE</code> that is a halt on ETH withdrawals, with the money sitting exactly where it should be.</p>
<p>The failure mode here is inverted. An ordinary circuit breaker is dangerous because it might not trip. This one is dangerous because it does, and it trips precisely when everything is fine.</p>
<p>And the cost of a false alarm is comparable to the cost of a miss: halted withdrawals at an exchange are not an inconvenience, they're an incident. Which means the breaker's inputs sit on the critical path, not in some auxiliary query.</p>
<p>The entire public conversation about proof of reserves, meanwhile, is about an exchange hiding addresses or borrowing funds. That is, about intent. About an enumerator losing its own addresses with no intent whatsoever, I found nothing.</p>
<p><strong>What catches it.</strong> Separate two different questions that look like one: "which addresses are ours" and "where are we currently accepting deposits". The second is filtered by activity. The first, never.</p>
<hr />
<h2>10. An error in the customer's favour is harmless</h2>
<p>It's harmful because it switches off the channel through which such errors are found.</p>
<p>An asset carries two numbers side by side. <code>precision</code>: how many places we count to. <code>decimals</code>: how many places it has on the rail. Only the second was ever reconciled against the chain.</p>
<p>And <code>precision</code> is one number per asset, while <code>decimals</code> differ across rails for one and the same asset. USDT was configured with a precision of 2 against six decimals on ETH, TRC20 and TON, and eighteen on BSC. USDC, the same 2 against six. ETH, 8 against eighteen.</p>
<p>The fee is rounded down by <code>precision</code>. So a fee below a cent became zero. A shortfall nobody knew about.</p>
<p>And nobody could have. A monetary rounding defect is normally found quickly, because somebody turns up and complains. Here there was nothing to complain about: the error ran in the customer's favour, and the feedback loop an engineer implicitly relies on had been switched off by the error's direction.</p>
<p>The second consequence faced outward. The same number was handed to customers through <code>/assets</code> as the input precision. A screen that built its input mask from it would refuse 0.000001 USDT from a human: an amount the network moves without a murmur.</p>
<p><strong>What catches it.</strong> Derive precision as at least the number of places across the asset's rails, rather than maintaining it as a hand-kept list. And the general rule the whole item exists for: reconciliations and alerts have to be two-sided. A one-directional error will never raise a ticket.</p>
<hr />
<h2>What they have in common</h2>
<p>Every one of the ten is true. In the case I had in mind while writing the code.</p>
<p>"An idempotency key makes a retry safe." True if the server invented the key rather than whoever turned up. "A send error means nothing went out." True up to the network call. "Address case is a property of the network." True while a chain hosts one format. None of them is a lie. Each simply has a domain, and past its boundary it stops working silently.</p>
<p>And I tested each of them on exactly the case I'd written it for.</p>
<p>That isn't a figure of speech. Here's an example that isn't in the list above: a query looked up incoming payments by a metadata key that only two rails out of four ever wrote. On the other two the screen showed "received 0, last received: never" for money that had arrived and been credited. I checked on TRC20, where it added up. The check was real: by hand, with real money. And that is exactly why it showed nothing.</p>
<hr />
<h2>What I haven't proved here</h2>
<p>This is not statistics. Ten cases from one set of repositories is one sample, with one author and one set of habits. Some of the classes did turn up elsewhere afterwards: a case-sensitive comparison against an allowlist sits open in peatio, two lines away from where the same address is normalised; an unbounded fee at a gas payer produced five consecutive CVEs in a single payments library this year; parsing an amount with an arbitrary exponent is closed by CVEs in both Elixir and the Go standard library. But eight other people's repositories is not a sample you compute frequencies from either.</p>
<p>I have not verified that my fixes are correct. I verified that they catch exactly the case that broke. Those are different claims, and the second is the weaker one.</p>
<p>And the gate caught none of the ten.</p>
<p>Eight were caught by two things. Four by addition on a live staging environment and a run with real money. Four by a separate pass of a security scanner. The remaining two arrived differently: one from checking the code against our own written rules, the other I don't remember. Which is to say that finding a hole in the checks took a check that wasn't among them. What to do about that, I still don't know.</p>
<hr />
<p><em>First published at <a href="https://jeffreyjorgensen.dev/falsehoods?from=hashnode">jeffreyjorgensen.dev/falsehoods</a>. If your system rests on a sentence like one of these, send it to me: I will tell you what it costs when it turns out to be false.</em></p>
]]></content:encoded></item></channel></rss>