I use an IOGear 4x4 USB switch to share the Xim4 between PS3, PS4, 360 and XB1. Mileage may vary in terms of power draw of mouse and keyboard, which can cause Xim disconnects, so consider plugging a powered USB hub into the Xim for the m, kb, or both.
You might need to use one controller plugged into the Xim rather than two behind a switch. If you're sitting at a desk for m/kb play, the switch is somewhat redundant given you use the same USB cable for PS4 and XB1 and can simply swap the controller.
That said, I believe the next firmware release will support cross-controller support (DualShock 4 on XB1 and vise-versa), so that switch may not be needed at all. I'm sure someone else will verify.
Finally, perhaps remove the battery from the XB1 controller. Leaving it in can cause connection issues. It works fine without a battery, drawing power from the Xim via the USB cable.