You can't reliably tell; whatever mechanism you could use could be spoofed in
any case.

For most cases, you shouldn't care which window manager is running, so long
as you do things in an ICCCM-conformant manner. There are some cases in which
particular window managers are known to do things wrong; checking for
particular hints placed on the window by the window manager so that you can
sidestep the problem may be appropriate in these cases. Alternatively, it may
be appropriate to determine which window manager is running in order to take
advantage of specific *added* features (such as olwm's push-pin menus) in
order to give your program *added* functionality. Beware of usurping the
window manager's functions by providing that functionality even when it is
missing; this surely leads to future compatibility problems.

Some parts © 2009 Advameg, Inc.