Hey I need an ADS Server guru to try to figure out what is going on with our server and give us some advice as we are moving to a new server and want to make sure we get it right this time.
Right now we have an unlimited user ADS Server installed on a Windows 2008 Standard R2 server with dual XEON processors and Hyper Threading and 24GB ram. This gives us essentially 16 cores but we have noticed that no matter what we do, ADS will never use more than 8 of the cores (I am guessing because the computer is set up using NUMA which divides the memory up to work with the nearest processor). We are noticing that as the ADS service ram usage approaches the 12GB mark (has never quite got there yet) that performance goes in the toilet but the CPU usage never goes above about 12%.
Was ADS ever written to take advantage of NUMA? Should we just turn NUMA off or do we need to downgrade the server to a single physical processor? It seems like with the NUMA nodes enabled that the physical computer is basically divided up into two machines and ADS will only run using half the resources (8 cores and 12GB ram).
At peak times, that server is taking connections from about 500+ users spread out over 5 servers running remote desktop sessions. The database server just has ADS running on it so that is the only process (other than standard Windows stuff) and we would like it to be able to use ALL the resources on the server instead of half of them. There are about 2400 database connections, 95000 work areas, 29000 individual tables and 24000 index files open at the same time so there is a lot going on. The average operations per second is usually around 17,000/sec with peak times where it can hit 350,000/sec for a minute or so. I made the assumption that if you threw massive resources at ADS that it would use them all.
Please note, we have checked all other areas for potential bottlenecks but the existing Raid 10 SAN is performing very well and the network cards are all Gigabit that even during peak times are never going above more than 5-6%. Disk Queue Length is almost always < 0.5, Disk Active Time is < 10% most of the time, Network Utilization is < 2%, CPU Usage is 5-12%, Physical Memory has 8.7G free
At one point we tried moving the server onto a VM and had that configured with anywhere from 4 up to 24 cores and from 16GB to 32GB of ram and the results were disastrous. Things worked fine in a test environment with 50 or so users connected to it for a month but as soon as we switched it over to having everyone log on using the VM for the database server, ADS would run for a few hours and then start throwing errors all over the place and disconnecting people from the databases. The only way to recover was to stop the ADS service and restart it which would work for another couple hours and then die again. No matter how we had the VM configured, ADS would die a miserable death with plenty of system resources unused and still available. At that time we were told that ADS was never written to run on a VM so we gave up on that experiment.
So, now we are looking at a solution that would have all our remote desktop sessions run on a bunch of VM's but have the ADS server be a physical server with a high efficiency SAN attached but we need to know what to build into that server. Windows 2008 Standard R2 only supports 32GB of ram so we don't need more than that. We just aren't sure what to do in regard to physical CPU's and NUMA. I don't want ADS to only have access to half the resources on the machine but at the same time, if shutting NUMA off makes everything run dramatically slower because the memory banks are randomly far away from the core running a query, we don't want that either. Maybe we have to bump it up to 32GB and then it can at least use 16GB for ADS and leave the other 16GB for Windows to play with?
These machines don't come cheap and we basically have one shot at this. The clients don't like us experimenting with things that don't work as we are supposedly providing them with a "cloud" solution that should be stable and not kick them off every hour to reboot the database software. We have been using ADS for a decade now and really don't want to switch but if we hit the wall then we hit the wall. Is there someone out there who knows what to do in this situation or did we just hit the physical limit of the ADS software? I can hope that someone at SAP knows the best way to set up their own software? Please?