If your system can't allocate 3 gigs of ram to a process without it paging, it may not be worth setting the /3gb switch, but I think you'll still see the benefit.
what metric were you using to get that 2.5 gig figure?
As regards to how much virtual memory x3tc will use, it'll use up to its virtual memory size limit (but will never hit an out-of-memory error). So on a vanilla system, it'll use up to 2 gigs and then unload resources it reckons it doesn't need. On a 32-bit system with /3gb boot flag and modded exe, it'll use up to 3gb. On a 64-bit system and modded exe, it'll use up to 4gb. How much physical RAM that means it can use is up to the OS, of course
Basically, the game never throws data out of memory until it has to, which makes sense. Also, even if you do run out of RAM before the game runs out of virtual memory space, the OS will probably discard resource files before writable data, so while the game is technically paging (as in "the memory page is on the disk") it won't have had to
write anything