Troubleshooting.

As always, add sudo to these commands if your user is not part of the docker group.

Visibility

./ethd logs -f consensus and ./ethd logs -f execution or more broadly ./ethd logs -f <servicename> will show you logs. When the service has been running for a while, you may want to start at the end, like so: ./ethd logs -f --tail 50 consensus.

docker ps will show you a list of running containers and their mapped ports.

Stopping and starting, updating

./ethd stop stops all services, ./ethd up starts them. ./ethd restart does both. ./ethd update brings in updates to the clients, ./ethd update --refresh-targets does that while resetting docker tag targets in .env to defaults.

Targets, source build

There are build targets in .env which, for the most part, you need not touch. For Lighthouse in particular, if your CPU is older than Broadwell (2014), you'll want the latest target, not latest-modern. Change LH_DOCKER_TAG=latest in .env and run ./ethd update, then ./ethd up.

If you want to build from source, change to Dockerfile.source instead of Dockerfile.binary for that particular client. The default source targets use the latest released tag on github; adjust to whichever tag or branch you want to build from.

Remove all traces of the client

This project uses docker volumes to store the Ethereum PoW and PoS databases, as well as validator keys for the validator client. ./ethd terminate will remove all volumes.

If you prefer to do that manually, you can see the volumes with docker volume ls and remove them with docker volume rm, as long as they are not in use.

This can be useful when moving between testnets or from a testnet to main net, without changing the directory name the project is stored in; or when you want to start over with a fresh database. Keep in mind that synchronizing Ethereum 1 can take days on main net, however.

Caution If you are removing the client to recreate it, please be careful to wait 15 minutes before importing validator key(s) and starting it again. The slashing protection DB will be gone, and you risk slashing your validator(s) otherwise.

Using Eth Docker with a VPN on the node

VPNs typically need IP addressing in the RFC1918 (private) range, and docker by default will utilize the entire range, leaving the VPN to not find a free prefix.

This can be solved by changing the configuration of Docker to use only a portion of RFC1918 for its address pool.

Interacting with Docker directly

./ethd cmd runs docker-compose or docker compose, depending on the compose version, and will use sudo as required. You can also run compose commands without using the ./ethd wrapper.

docker-compose stop servicename brings a service down, for example docker-compose stop validator.
docker-compose down will stop the entire stack.
docker-compose up -d servicename starts a single service, for example docker-compose up -d validator. The -d means "detached", not connected to your input session.
docker-compose run servicename starts a single service and connects your input session to it. Use the Ctrl-p Ctrl-q key sequence to detach from it again.

docker ps lists all running services, with the container name to the right.
docker logs containername shows logs for a container, docker logs -f --tail 500 containername scrolls them.
docker-compose logs servicename shows logs for a service, docker-compose logs -f --tail 500 servicename scrolls them.
docker exec -it containername bash will connect you to a running service in a bash shell.

You may start a service with docker-compose up -d servicename and then find it's not in docker ps. That means it terminated while trying to start. To investigate, you could leave the -d off so you see logs on command line:
docker-compose up consensus, for example.
You could also run docker-compose logs --tail 100 consensus to see the last logs of that service and the reason it terminated.

If a service is not starting and you want to bring up its container manually, so you can investigate, first bring everything down:
docker-compose down, tear down everything first.
docker ps, make sure everything is down.

If you need to see the files that are being stored by services such as consensus, validator, execution, grafana, &c, in Ubuntu Linux you can find those in /var/lib/docker/volumes. sudo bash to invoke a shell that has access.

HERE BE DRAGONS You can totally run N copies of an image manually and then successfully start a validator in each and get yourself slashed if the built-in database lock fails. Take extreme care.

Once your stack is down, to run an image and get into a shell, without executing the client automatically:
docker run -it --entrypoint=/bin/bash imagename, for example docker run -it --entrypoint=/bin/bash lighthouse:local.
You'd then run Linux commands manually in there, you could start components of the client manually. There is one image per client.
docker images will show you all images.