Everything started in October 2020 when we decided after weeks of discussions to finally get involved with the Lightning Network by running our very own Routing Node.

The objective of this series of articles is to tell our story, our struggles, and yes, our successes.

Inspired by MicroStrategy’s move to invest part of their treasury into bitcoin, we lead the process for our student association, KryptoSphere, to do the same and use our funds to participate in the inevitable expansion of the Lightning Network. From the very beginning we established some ground rules and objectives for the future of the KryptoSphere Node (pronouns she/satoshi):

• The most important metric to consider should always be the volume/number of payments routed rather than the amount of fees collected. Indeed, as a student project but most importantly as a group of friends strongly believing in the importance of the LN, the aspect of profitability associated with running a routing node was not our primary concern.

• We wanted to do our part in developing the network. For this, we needed to collect valuable data that could help others like us (LN newbies) to understand the underlying technical and economical challenges associated with running a node.

• Experimenting with everything at our disposal to make our node the most efficient node in Europe (ie. Routing as much as possible, with as little capital as possible and with as less operating costs as possible).

• Enter the Top 100 in the BOS score list.

• Have Fun (and hopefully not staying poor).

In the first part of this series, we’ll touch on our humble beginnings. The early struggles of setting up a secure node (yeah our router almost burned), the constant quest for liquidity (sometimes I wonder if you can drink a lightning channel because someone must be drunk), and the valuable lessons we learned along the way (Choosing and connecting to performing nodes, establishing a sustainable fee policy, the Girard Ratio, etc..). Finally, we’ll end with an overview of our node’s performance since October. This part will be illustrated with some of the data we’ve collected and that Fanis transformed into magnificent charts. Enjoy!

The choices we’ve made for our node: hardware, security, and the cost of complacency (Fanis)

When it comes to running a Lightning routing node and hoping not to blow everything up, a few, accessible skills are required. Especially, it seems a bit foolish to try running a routing node without some knowledge of how UNIX (and, of course, the Lightning Network) works. In this part of the article, we’ll go into some details about the steps we took to improve the security and efficiency of our routing node. Please keep in mind that:
- we’re a bunch of amateurs.
- in such a domain, there is no such thing as “being done”. It’s always a work in progress!

Collecting the hardware

A Lightning node begins with the hardware. We kept it simple here: Raspberry Pi4 with 4 GB of ram and 1 TB SSD. If you’re looking to build your own node, we strongly recommend the use of an SSD, which is totally worth the extra money compared to an HDD, especially during the initial sync.

After having some trouble with the node unexpectedly “shutting down”, we added a Flirc aluminum case to deal with the heat (from a mean temperature of 70°C before to 45°C after), and changed the power supply to the official one for good measure. We also added some scripts to periodically log the health status of the Raspberry Pi in a file on the SSD, so that we could investigate if the issue was to happen again — which it never did.

Add some security

Another thing to focus on when it comes to Lightning is security. Lightning currently requires you to hold your funds in a hot wallet, so it is very important to make this wallet as secure as possible. Because secrecy is part of security, we won’t go into too many details regarding our password generation procedure, but here are a few things we wanted to share.

We opted for a dice-generated password. Using a very long list of words and a few dices, we generated various passwords (for instance, the one for Lightning Terminal, and the one that unlocks access to the Raspberry Pi hosting our node). With a reasonably large amount of words per password (a compromise between entropy and ease of use), a high enough frequency of changing passwords and fail2ban set up, we believe it provides us with a sufficient level of security given the funds at stake. For those who are not familiar with it, fail2ban is a very useful tool that will protect your system from brute force intrusions (esp. over SSH) by “banning” any IP that tries to log in too many times with a wrong password. More importantly, we also limited access to the Raspberry Pi using RSA key authentication, which means anyone who wants to connect needs to have the appropriate key. We also deactivated root access over SSH.

When we decided to change the password, we did so over a secured call by rolling dices and vocally sharing the results of the rolls. This way, nothing is ever written digitally, and as we only vocally exchange numbers, any intruder would also need to know which word list we’re using. We believe it is enough security given the funds at stake in our case. We highly encourage you to weigh the pros and cons and adjust the security level to your needs.

Backups

One of the key things to understand is that restoring a wallet’s balance on Lightning isn’t as straightforward as with a Bitcoin wallet (if the recuperation process with a Bitcoin wallet’s seed and derivation path can be called “straight forward”). Here, in case you lose access to your node and need to get your funds back on a new wallet, your seed phrase won’t be enough. You also need what is called a channel backup to correctly close all your channels and get your funds back on Bitcoin.

The channel.backup file that is produced by lnd is, therefore, something you want to backup safely. It was designed to be sent over to servers without major implications in terms of privacy, so a good practice is to send it to at least one remote server every time it changes. We knew it. Yet, we still took too much time to act accordingly, and it could have cost us a lot of money.

One night, a big thunderstorm raged in the area where our node is located. 

Eventually, the power and the internet went out. On the next morning, when everything went back up, we could not reach our node via ssh. Remember, we had a few tens of millions of sats sitting on this node, most of it already engaged in channels. 

The loss of our channel.backup file would have meant the loss of our funds.

Luckily, we quickly discovered that the SSD and the SD card were not damaged. The Raspberry Pi itself seemed okay, except for the ethernet connection that didn’t seem to work. And then it occurred to us: thunder had literally burnt the ethernet ports on the box that were connected to a device. Or, to put it differently, the ethernet port of the box played its role perfectly, sacrificing itself so that the over voltage wouldn’t reach the device on the other side. 

Oof. 

Funds were SAFU, and plugging the ethernet cable in another port solved the issue.

TL;DR: our *Lightning* node nearly got destroyed by *thunder* at a time when we didn’t have any backup for our funds because we were lazy ass bitches.

So the very same day, we manually exported the channel.backup file to a remote server, and started writing an automatic script that would export the file to the remote server whenever it changes. On the evening of the same day, the script was written and running. (link to the scripts at the end of the article)

Routin’Omics 101: Managing liquidity, choosing the right peers, and defining a sustainable fee policy (Guillaume)

This part will walk you through the decisions we’ve made concerning three distinct parameters of a routing node: Channels, Fees, and Liquidity.

In the process of documenting the activity of our node, I’ve identified 3 different phases that match the progress we’ve made since we began this adventure.

The first one I decided to name: The Savage State.

Yes, the savage state was the very beginning of the KryptoSphere Node. We had pretty much no idea of what we were doing, and just like savages we paid the cost of not understanding the environment that surrounded us.

Liqudity and peers

The first step of the process was to establish a list of potential nodes with who to connect. But how can you choose a list of viable candidates? Your goal is to route as much as possible, so you obviously want to choose a big, healthy, and well-connected node. Or do you ..? It goes without saying that all rational actors in the network will try to adopt a profit-maximizing strategy, but remember, our initial goal was not to make a profit/generate a form of passive income. So, we tried to identify “undervalued” nodes with the most routing potential that were not already flooded with liquidity.

For that, we used our knowledge of the industry and the few tools at our disposal. Having followed the evolution of LN since 2018, we had a pretty good idea of what to look for in terms of growing companies operating a Lightning node. (if not already you should subscribe to both LightningLabs and LNMarkets newsletters to never miss what’s happening on LN).

Given the anticipation for a new bull market and the probable subsequent rise in fees on L1, we decided to focus on exchanges, betting on the trend of LN integration that would drastically reduce fees for users. We figured nobody would accept sky-high fees for a $20 transaction like in 2017. We also looked for emerging companies on LN that could see a meaningful increase in the use/sale of their product in the coming months (in retrospect it was inevitable).

With the help of 1ML, the Acinq Explorer, and one of the brightest mind in Bitcoin a.k.a Romain Rouphael (LNMarkets Co-Founder) we settled on the following companies :

• LNMarkets: It was a no-brainer really, betting on the growth of LNM we chose to allocate 4,231,948 sats.

• Southxchange: We identified Southxchange on Twitter and looked at their stats on 1ML. We allocated 5,302,384 sats. They are really well connected, have a lot of liquidity, but most importantly we believe that their geographical location is very promising. (BTC is booming in Argentina)

• Tippin.me: We all knew Tippin prior to building the node, we know it is well known in the community and has a good chance of seeing a lot of volume. 5,134,843 sats allocated

• Fold: Don’t think I have to justify why we chose Fold. Even though it was an already well-connected node, we believed that their potential for exponential growth justified allocating 5,037,248 sats. Spin the wheel!

• BCash_Is_Trash: The name by itself was enough to convince us. With 6,103,938 sats allocated, this node was very high on the BOS score list, well-connected, and very healthy overall.

• WalletOfSatoshi.com [2]: Well respected LN Mobile wallet, underrated at the time in our opinion. Justifying a 5,024,814 sats allocation. [NOW CLOSED]

• Bitstamp: They had just integrated LN earlier in the year, based on our hypothesis we thought this node will get a lot of volume. 2,194,284 sats. [NOW CLOSED]

• OpenNode: As an underrated competitor to the already crowded BTCPay node, we believed OpenNode had a lot of potential by facilitating merchant adoption. 3,894,562 sats. [NOW CLOSED]

• JohnOnChain: A fellow Frenchman that was already routing a lot and accepted to open a channel with us. 1,957,452 sats. [NOW CLOSED]

The River channel opening marked the beginning of the golden-era for our node. 

Much routing, sats go up. I’m a long-time admirer of River, I think they have a unique positioning and customer experience. They were one the first exchanges to see the potential and superiority of the LN for their clients, so, I tried to contact them directly on Twitter to convince them to open a channel with us and they did!

We also opened a 5 million sats channel with Zebedee.

Zebedee is a Lightning payment processor, with a focus on Lightning-enabled games, where users can win real satoshis for example. Our decision to open a channel with them was led by the love of Fanis for Bitcoin and Lightning infused games, some of which leverage Zebedee’s solution to send payments to players. We also thought it would be a great peer to connect to, with a lot of traffic but still a bit “under the radar” (compared to big nodes such as Bitrefill for example).
Upon opening the channel, we found out that the fee on their side was quite big (1000 ppm). Suspecting that it was only because they might have some automatic policy for unsolicited channels, we contacted Zedebee and they (thanks to Big Boss André) quickly set a lower fee to 1ppm. Moral of the story, watch out for your peer’s fees!

Something to watch out for is the Channel Acceptor parameter: While we initially set up ours to 1M sats (something about the median size of channels I’m told) we later decided to lower it to 500k in case we missed out on volume generating channels.

Fee policy

The first choice we’ve made when it comes to fees was to reduce both the base fee and variable fee to 0.001. We probably thought we were geniuses at the time, thinking we would route a lot more payments this way. What happened, as indicated by the name of the era, is that we got brutalized.

People used our node to make huge payments and drained all of our liquidity in a matter of days.

Some valuable insight we derived from this was that if you want to route a lot of payments but not drain your liquidity, you can set your base fee to 0 and raise your variable fee so that it will disincentivize big payments. As we raised our variable fee to 100ppm, we witnessed a steep decline in routed volume, so we adjusted down to 20, and again after that to 5, which has been our fee level for quite some time now. We plan to raise this in the future but as of now, it seems to bear good results.

Rebalancing channels

• Circular Rebalancing: Using Balance of Satoshi by Alex Bosworth we used this type of off-chain rebalancing very early on. Because you need to pay a fee for each “hop” it can rapidly become quite expensive to rebalance your channels this way. However, in a high on-chain fee environment like today, it might be cheaper to use circular rebalancing. In this case, we recommend that you check the different “bos routes” proposed and choose only the cheaper ones. (https://github.com/alexbosworth/balanceofsatoshis)

• Loop: Developed by Lightning Labs (of course), Loop is a tool that allows you to rebalance your channels using the Bitcoin blockchain. More info here: https://lightning.engineering/loop/ . We mostly used Loop to rebalance our channels because at the time fees were quite low on the Bitcoin blockchain. By using the command line you can avoid using the “fast mode” enabled by default and save some fees. Most of our Loops cleared at 3 sats/Vbyte or cheaper.

NamsBreakEvenPoint

In order to measure how effective you are at running a routing node, you can use the NamsBreakEvenPoint. This is the point where your Total Balancing Cost is covered by your Total Routing Fees. For the first phase of our experiment, we decided to focus on volume rather than fees but we will definitely use it in the future. It is absolutely necessary to measure if the cost of rebalancing your channels is not higher than the fees you’ve collected. It would be shameful to disclose our current Nams ratio, so I will spare you this pain…

Then we kinda lost motivation, enter The Winter blues Era

Yeah… We did nothing for an entire month and left our beloved node to rot. As we witnessed the volume of sats routed plummet to zero after our last adjustment, we lost our mojo. As far as we were concerned, we failed and our node was “dead”. Or so we thought…

Turns out, we are idiots. The node was displaying no volume due to a scripting error and our strategy was actually proving to work very well! I’ll let Fanis explain to you what happened. (totally his fault btw, you can hold him accountable for the evident lack of content in this part. Should’ve known not to trust the Greeks with money since 2011…)

Read The Fucking Manual

To track the evolution of the volume of payments routed by our node, we designed a basic script that uses lnd fwdinghistory command to retrieve forwarded payments information and display it in a nice table. But the number of forwarded payments stopped to increase at exactly 100. Eventually, we figured out that the fwdinghistory command defaults to displaying only the first 100 forwarded payments at a time. Running the same command but specifying that we wanted to see the 50,000 last forwarded payments, we discovered that our node had actually been routing a lot of new payments (around 800 payments) in the past months. We just couldn’t see it.

There’s a quite simple, generalizable lesson here: when you use a tool, carefully read the documentation. The fact that the fwdinghistory command needs to be supplied with a max_events argument, and defaults to 100 events, was in plain sight all this time.

The Rebirth Phase

Once we discovered the scripting error, we regained confidence in our ability to become once again routing overlords. Some house cleaning had to be done before we could continue normal operations (ie. rebalancing some channels and re-evaluate our current strategy). To evaluate our current strategy we started looking at the way we managed our liquidity because while we were away, we noticed that some channels tended to balance themselves without our intervention while others remained completely unbalanced after a single large payment.

Liquidity Management

We learned very fast that maintaining a routing node was largely dependent on how you can manage your liquidity, because without liquidity your channels are useless.

One thing that you could use to reduce the burden of liquidity management is the AutoRebalanceQualityRatio.

AutoRebalanceQualityRatio

We used this ratio to determine if a channel can rebalance itself in order to reduce rebalancing costs over time. The closer the ratio is to 0, the better. It could become a viable strategy for routing operators looking to save on rebalancing costs to only allocate funds to channels with a ratio close to 0.

For example: While LN Market is a performing channel in terms of routed volume, with a ratio of 0.30, some maintenance in the form of rebalancing costs might be needed to keep routing. On the other hand, a channel like Zedebee, comparable in terms of routed volume, has a much better ratio of 0.037 (!). Now, keep in mind that this is an average over time and there are other factors to consider when evaluating the efficiency of a channel (a channel with a ratio of 1, like River Financial, can still route a lot of payments, you’ll just have to rebalance it regularly).

Routing Analysis

Something that we also started to monitor is the Girard Ratio (named shamelessly after yours truly). The Girard ratio is a measure of the overall routing efficiency of a node (or individual channel) by comparing how much Bitcoin you locked on LN to how much you’ve actually routed for this amount of BTC. The higher the number, the better. For example, a ratio of 3 would indicate that for 1000 sats allocated to a channel, this channel routed over 3000 sats. Much wow, routing go up!

Girard Ratio

The current Girard Ratio is 5.97 indicating that our node has routed on average 6 times the amount of Bitcoin we allocate for it.

For the Girard Ratio of a channel in particular, things can be a bit different. The goal is to be able to compare two channels on their respective Girard ratios. We believe a peer that receives a lot of payment (eg. with a big “forwarded out” amount) should be treated as equally important as a peer that sends a lot of payment (eg. with a big “forwarded in” amount), simply because they are just the two sides of the same transaction that passes through our node. Therefore, it seems appropriate to take the sum of both “forwarded in” and “forwarded out” parts as the “TotalForwardedVolume”, and divide by the number of sats we locked into this channel.

You can also use the ratio to compare channels, for example, LNMarket has a ratio of 8.85 while Zebedee has a lower ratio of 3.02. This goes to show that while Zedebee has more auto-rebalancing capabilities, LNMarkets is just the king of routing!

During this period from the end of January to the present date, we’ve reached a few milestones for the KryptoSphere Node that we’d like to share.

We are still not sure of what justified such a rapid acceleration in our routed volume since we had not interacted with the node in over a month (Bad Operators). We can only speculate that it was in part related to the extent of the Bitcoin Bull run. I’d also like to believe that it was a consequence of the strategy we chose for the node. It was designed specifically to profit the most in case of a severe spike in transaction costs on Bitcoin.

Milestones that make us proud :

• Over 1 BTC routed ($50 000 in value)

• Over 1000 routed payments (1008)

• 854 sats collected (Oof, but the goal is not revenue)

• 50M+ sats capacity for the KryptoSphere Node

• No MAJOR fuck-ups or loss of funds (even though we were asking for it)

• Discovered valuable insights (Girard Ratio, NamsBreakevenPoint, Fee strategies, liquidity management…)

• Created meaningful relationships with LN people and companies that will allow us to further experiment with our node.

• We learned so much, going further than ever before down the rabbit hole!

Thank you for reading about our Lightning adventures. A second part will be published in the coming months as we continue with our experiment. We welcome any feedback and/or idea for Part 2. You can find all the data we presented here.

We’d like to thank Sohomang, Zied Hagui for the photoshop genius, and Romain Rouphael for helping us make this project a reality, and to KryptoSphere for their trust.

Here are a few scripts we’ve made for the maintenance of our node and to monitor it. They’re not perfect, but feel free to take what’s useful for you:

• Casey is a tool that for now displays a table with your forwarding stats, kinda like the bos tool does. We intend to expand its usage in the future (and rewrite some part of the existing code, which is really far from perfect): https://github.com/fanismichalakis/casey

• Script to automate channel.backup exports to a remote server every time the file changes (requires systemd services): https://gist.github.com/fanismichalakis/5d3e77601d357362b54c797638a9597b

• Script that auto-unlocks the lnd wallet on the node booting up. Very useful to reduce the unavailability period when the node goes offline and comes back online for some reason (such as temporary internet failure for example). You can find some details on how to set it up for your node here: https://stadicus.github.io/RaspiBolt/raspibolt_6A_auto-unlock.html