Graceful Exit Guide

(This guide was originally published by Jens Heimbürge in this forum thread. Please read the thread in order to find the most recent updates and changes).

Warning: Read this carefully before you start
Please read the following information carefully and ask any questions your might have on this forum thread before executing graceful exit.

• Your node will be disqualified if during graceful exit, more than or equal to 10% of transfers fail. In such a case, this will then also result in a failed graceful exit and loss of the held back amount on the satellite you were running the graceful exit for. 
• Once graceful exit is started, there is no way to stop or cancel it. If graceful exit fails even at 99% of completion, you will not get any of the held back amount. Think about the consequences twice before you call it.
• All payouts are subject to the minimum payment threshold, include final payout after Graceful Exit, if a SNO wants to receive their final payout and it isn't above the threshold they need to opt in to zkSync if they have not done so already.

Requirements:

• The Storage node has joined the satellite more than 15 months ago (this requirement was temporarily reduced to 6 months, but the required wait time may be changed as needed).
• The Storage node is healthy and hasn’t lost data at the time of requesting a graceful exit. Disqualification during graceful exit is possible.
• The Storage node has no significant downtime. If the storage node is offline for a few days while running graceful exit, it will fail.
• The Storage node has no bandwidth caps. Graceful exit doesn’t take into consideration any bandwidth caps you may have from your ISP. It will go on with the transfer even if the bandwidth cap is exceeded. You are not allowed to take the storage node offline or block the traffic, as is would result in disqualification. You are allowed to call graceful exit in the middle of the month to split the graceful exit traffic over 2 months. You are allowed to call graceful exit one satellite at a time to split the traffic even further. Make sure you have no other processes running using significant bandwidth while you are executing graceful exit, to minimize the chance of failure. You may even consider reducing your allotted storage space temporarily to a level that would make the node appear to be a full and prevent it from accepting new pieces while executing the graceful exit to further reduce bandwidth usage.
• Graceful exit traffic is unpaid. If you finish graceful exit successfully, you will receive the held back amount corresponding to the satellite(s) you executed the exit for.

Start Graceful Exit

Are you sure you want to start graceful exit? Did you read the information above? Do you understand that graceful exit can’t be canceled once initiated?

Here is an example how you call graceful exit from the command prompt (Linux):

docker exec -it storagenode /app/storagenode exit-satellite --identity-dir /app/identity

Please be aware that by starting a graceful exit from a satellite, you will no longer be allowed to participate in repairs or uploads from that satellite. This action can not be undone. Are you sure you want to continue? y/n : y

Domain Name                      Node ID                                             Space Used
satellite.stefan-benten.de:7777  118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW  1.8 TB
ap1.storj.io:7777   121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 37.9 GB
us1.storj.io:7777  12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S 38.7 GB
eu1.storj.io:7777 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs 0.8 TB

Please enter a space delimited list of satellite domain names you would like to gracefully exit. Press enter to continue: satellite.stefan-benten.de:7777 ap1.storj.io:7777 us1.storj.io:7777 eu1.storj.io:7777

For Windows GUI users, the exit command should look as follows, but be sure to point to your own identity and config directories (in cmd.exe):

"C:\Program Files\Storj\Storage Node\storagenode.exe" exit-satellite --identity-dir "C:\Users\USER\AppData\Roaming\Storj\Identity\storagenode" --log.output stderr

Please be aware that by starting a graceful exit from a satellite, you will no longer be allowed to participate in repairs or uploads from that satellite. This action can not be undone. Are you sure you want to continue? y/n : y

Domain Name                      Node ID                                             Space Used
satellite.stefan-benten.de:7777  118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW  1.8 TB
ap1.storj.io:7777   121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 37.9 GB
us1.storj.io:7777  12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S 38.7 GB
eu1.storj.io:7777 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs 0.8 TB

Please enter a space delimited list of satellite domain names you would like to gracefully exit. Press enter to continue: satellite.stefan-benten.de:7777 ap1.storj.io:7777 us1.storj.io:7777 eu1.storj.io:7777

You can exit the satellites one by one or all at the same time. The satellites you requested graceful exit for will need a few hours to create the list of pieces that need to be transferred. Don’t expect high traffic immediately.

For multiple nodes owners

If you used multiple nodes on the same host, you have changed the server.private-address option in your config file. You must use this option in the exit-satellite and exit-status commands.

For example, if you changed it to server.private-address: 127.0.0.1:7779 and your other identity is located in "C:\Users\USER\AppData\Roaming\Storj\Identity\storagenode2", then the exit-satellite command will look like:

"C:\Program Files\Storj\Storage Node\storagenode.exe" exit-satellite --identity-dir "C:\Users\USER\AppData\Roaming\Storj\Identity\storagenode2" --log.output stderr --server.private-address 127.0.0.1:7779

During Graceful Exit

You can watch the status of graceful exit as follows (example output from a test satellite):

docker exec -it storagenode /app/storagenode exit-status --identity-dir /app/identity

Domain Name     Node ID                                             Percent Complete Successful Completion Receipt 
127.0.0.1:10000 12fbck97kqEGbWPu673CpeyrXavtqgVriyv9pCfL3mpw3yz2zN9 0.00%            N          N/A

From the cmd.exe (Windows)

"C:\Program Files\Storj\Storage Node\storagenode.exe" exit-status --identity-dir "C:\Users\USER\AppData\Roaming\Storj\Identity\storagenode" --log.output stderr

Domain Name     Node ID                                             Percent Complete Successful Completion Receipt 
127.0.0.1:10000 12fbck97kqEGbWPu673CpeyrXavtqgVriyv9pCfL3mpw3yz2zN9 0.00%            N          N/A

For the second node with identity "C:\Users\USER\AppData\Roaming\Storj\Identity\storagenode2" and server.private-address: 127.0.0.1:7779:

"C:\Program Files\Storj\Storage Node\storagenode.exe" exit-status --identity-dir "C:\Users\USER\AppData\Roaming\Storj\Identity\storagenode2" --log.output stderr --server.private-address 127.0.0.1:7779

If you get the output No graceful exit in progress. this means graceful exit didn’t start because of the minimum node age requirement. In the storage node logs you will find additional information such as: node is not yet eligible for graceful exit: will be eligible after 2020-04-02 01:18:23.910919 +0000 UTC.
In case of a crash, power failure or other outage during graceful exit, please get your storage node back online. Graceful exit will continue.

Concurrency

By default, the storage node will transfer 5 pieces at a time. You can increase that via config, but be careful that you are not using values of concurrency so high that they may cause too much bandwidth usage. It is best to stick with the default value for concurrency if you are not sure of the implications. Note that the default concurrency is 5, which should translate to about 20MBit/s upload traffic: graceful-exit.num-concurrent-transfers: 5

Finish Graceful Exit

At the end you will get this output: (sample output from a test satellite)

root@kali:~# storagenode exit-status

Domain Name     Node ID                                             Percent Complete Successful Completion Receipt 
127.0 0 1:10000 12fbck97kqEGbWPu673CpeyrXavtqgVriyv9pCfL3mpw3yz2zN9 100.00%          Y          0a473045022100da86329cfb4f5bb16f0702c1d073c3a8b54787311b54855bcf01a8e245250040022003ef911b3b2b2bea86ba34cd4927223f2718cd35c3b7de7cc030cd3a8ce4959a1220db55bd9fa76e8938be5a7a25c970d48bde19936e269dcf69a3ab9fa41b5486001a207508f9a6138cdc4089ea075f1553736d472cb1d3afa4397496a8eb948d121200220c08abe5dcf0051086e6fefe01

In some situations, the storage node doesn’t need to transfer 100% of the data and might finish graceful exit with a lower percentage. As long as graceful exit was successful, you should get your held back amount along with the next regular SNO payout on Layer 1, if the owed amount is above the minimum payout threshold, or if it is opted in to zkSync on Layer 2. The “Completion Receipt” contains a signature from the satellite you exited and is your ticket to get the held back amount. Please keep your nodeID, each satelliteID and each completion receipt in a safe place. With this information your can open a support ticket if needed.