For years now I've been using and managing my Linux server over a SSH connection. Indeed, this entire website was coded up using vim over SSH. In the beginning Unix machines were primarily accessed via teletypewriter (tty's) and CRT terminals connected through a serial connection, and this tradition lives on in the plethora of terminal/console applications for *nix systems (unlike Windows systems, which are virtually impossible to administer from a console alone). So, I've been managing my Debian and Ubuntu systems for years now without ever firing up an X server, happily reading my mail with the proud but austere mutt, seeking tech advice on freenode using the ever clever irssi, haunting the occasional programming news group via the enigmatic but flexible slrn, and more recently downloading *nix ISO's over bittorrent with rTorrent. All managed with the indispensable terminal multiplexer screen, which is essentially a window manager for terminals, with the added goodness of letting you disconnect and reconnect to sessions.
Of course, I've tinkered with tunneling X through SSH, tunneling VNC, the somewhat faster tight VNC, and I even tried the XRDP project which aims to provide a Windows compatible RDP server (remote desktop) for *nix systems. In the past I'd combine one of the above with a bare minimal window manager like ratpoison or evilWM, and I'd achieve somewhat usable speeds. Certainly tight VNC isn't a bad option, and it can even provide for detachable and re-attachable X sessions, but nothing has really lured me away from screen till now.
Enter NoMachine's NX. All I can say is I am simple stunned. The speeds are phenomenal. Indeed, nothing really prepared me for this kind of remote X goodness. I have a Gnome desktop running, and I am happily rediscovering X Windows and the Linux desktop after a long absence. As much as I love ncurses and screen, I think it would be foolish to deny that certain applications can benefit from a graphical interface. So, If you need remote access to your Linux desktop, I would definitely check it out. There is an open source offshoot called FreeNX, which is what I currently have installed on my Ubuntu system. NoMachine is also kind enough to offer their server with few strings attached, and clients are available for virtually all platforms.