The nginx config provided in the Docker installation part contains everything needed for nginx. If you are installing lemmy directly on the machine you may need to use different upstreams.
The websocket part is basically the
# proxy common stuff
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
part in the nginx config on that page.