Optimizing and Improving Opensimulator Performance

Optimizing and Improving
OpenSimulator Performance
Justin Clark-Casey
OpenSimulator core developer
In this presentation
The System Architecture
Components and interactions
Potential bottlenecks
Rules of thumb for good performance
Measuring and Testing
What the heck is going on?
The Conference
400 connections for the keynote
Why listen to me?
OpenSimulator core developer and consultant for 7+ years.

Have carried out grid performance analysis for commercial
and education clients.
Have made many performance improvements, especially
for conference this year and last.
ghosts branch to soon be integrated.
But even I'm not an expert on everything
Performance is a very complex topic in this architecture.
Many different network & server arrangements.
The System Architecture
Talking about a Hypergrid-enabled grid installation.
Though much also applies to standalone.
Much also applies to a non-Hypergrid installation.
Assume you are already familiar with fundamental

system concepts
Standalone, grid, Hypergrid.
Assets, inventory.
HTTP, TCP, UDP.
Grid Components
Viewer
Simulator
HTTP
UDP
HTTP
Grid Services
HTTP
Simulator
HTTP
HTTP
Public
Private
Hypergrid Components
Viewer
Home Simulator
UDP
UDP
Foreign Simulator
Home Services
Foreign Services
Bottlenecks
In general four aspects
Network
Processing (CPU/GPU on viewer side)
Storage
Software
for three components
Simulator
Grid Services
Viewer/Client
Network Bottlenecks
Viewer Simulator, Services
UDP: need low latency
HTTP: need high bandwidth
Rule of thumb: try at least for 500 kbit per connection
pingtest.net
Simulator Simulator same grid
HTTP: Low latency
Install Install via Hypergrid
HTTP: Low latency, good bandwidth
Processing Bottlenecks
Viewer
More prims, more avatars, higher load.
GPU/CPU limiting factor
Simulator
More prims, more avatars, more scripts, higher load.
CPU limiting factor in the first instance
Recommended minimum 2 cores per simulator

Rule of thumb: 1 core per active region
One region vs multiple regions per simulator.

Software inefficiency (more on that later).
Storage Bottlenecks
Services
Asset storage dominates everything.
Deduplicating asset service in future release
All assets immutable so deletion extremely tricky.

Or use SRAS today [SRAS].
Faster storage is better but probably not critical.
Simulator
Allow as many assets to be cached as possible.
Software Bottlenecks
Viewer
Services
I know little about this.

Maybe inefficient C# Webserver though debated.
Simulator
Free threads for all! approach inefficient.
Improvements for conference.
Mono vs Windows .NET.
Offload GetTexture/GetMesh to direct service calls [DIRECT SERVICE]
Handling bad foreign installations in the Hypergrid.
Measuring and Testing
Every install can be different.
Very large number of moving parts
More operating system than app/webserver.
Identifying and reproducing an issue
Is critical to actually fix it.
Often much >50% of the work.
Measuring
Generations of stat systems...
Dominant is now show stats all.
Console commands on simulator and services
Record raw numbers and moving averages
Facility to periodically record to file.
Measuring
Simulator (root)# show stats all
clientstack.keynote1.ClientLogoutsDueToNoReceives : 0
clientstack.keynote1.InboxPacketsCount : 0, 0/s, 0/s
clientstack.keynote1.IncomingPacketAsyncRequestsWaiting : 0
entitytransfer.keynote1.InterRegionTeleportAttempts : 0
entitytransfer.keynote1.InterRegionTeleportCancels : 0
entitytransfer.keynote1.InterRegionTeleportFailures : 0
httpserver.5000.HTTPRequestsServed : 0 requests, 0 requests/s, 0 requests/s
scene.keynote1.RootAgents : 0 avatars
scene.keynote1.ScriptLines : 0 lines/sec
server.memory.ProcessMemory : 143.59 MB
server.network.HTTPRequestsMade : 22 requests, 0 requests/s, 0 requests/s
server.processor.CPUPercent : 0 percent
More information at [Stats].
CPU Sample Graph
Agent Time Graph
STP Work Items Waiting
Graphing
In my own tools repository. See [GRAPHING]

reference.
Works on stats record start log files.
New this year.

Format may change in the future.
Primitive graphs
No outlier filtering, time series alignment, yet, etc.
But can still be useful in diagnosing issues after the fact.
Testing
General network
iperf Tool for performing TCP/UDP throughput and

latency tests [iperf]
Services
Get texture test code at [TEXTURE LOAD TEST].
Otherwise not much yet.
May be able to use general test tools such as Siege

[SIEGE].
Testing
Simulator
pCampbot [PCAMPBOT]
Program to manage connections of many bots from

a single console.
Different behaviours (teleport, walk around, etc.)
This is a major tool I use in conference performance

testing.
Will continue to make behaviour more like real

viewers.
The Conference
Getting 400 connections in the same space is a

huge challenge.
Load grows exponentially not linearly.
Very rarely been done up to now.
Considerations
Can other parts of the grid take it (e.g. services)?
Do we have the bandwidth?
Getting people to login early and hence cache assets helps.
Do we have the CPU?
24 core system for 28 regions
Yeah, we're pushing it.
Varregion vs. 4 neighbouring regions
Fault tolerance
Code maturity
The Process
Regular Tuesday load tests supplemented by

pCampbot.
First locally then Amazon EC2.
Stats recording and analysis.

Reproduce problems seen in live tests with
pCampbot.
Changes to OpenSimulator code or config changes.
Repeat until exhausted.
Problems that Emerged
Needed to fix issues in/improve pCampbot.
Needed to create graphing and analysis code.
So what do you need? Besides a miracle.
Stats, lots of stats.
Reduce UDP traffic where possible.
OpenSimulator trying to do too many things at once.
Super thread happy.
Lots of context switching.
Holds up other work.
Physics
Not a major issue in the keynote scenario.
Configuration tweaks
[InterestManagement] in OpenSim.ini
RootRotationUpdateTolerance = 0.55
ChildReprioritizationDistance = 99999
Reduce UDP updates on avatar rotation.

Reduce thread work for neighbour sim observation.
ChildTerseUpdatePeriod = 4
Reduce UDP traffic from child connections.
Configuration tweaks
[ClientStack.LindenUDP]
enable_adaptive_throttles = false
scene_throttle_max_bps = 50000000
Observed bad behaviour under very high load with

adaptive.
400 mbits/s
client_throttle_max_bps = 125000
1 mbit/s
Software changes
Pure performance
Perceived performance
Process XML as stream not document.

Skip LLClientView queue for own mvmt packets.
Less thread happy
Sequential processing.
Appearance/Attachments
Some incoming/outgoing UDP handling.
Complex need to avoid one task blocking another.
Future Directions
Can probably make efficiency gains by further controlling

concurrency.
Simulator-wide job scheduler?
Distributed Scene Graph.
Simple component to maintain connections even if other

processes crash.
Allow risk of running more regions per simulator.
But improving single system efficiency easier.
Varregions
High concurrency challenges over a larger area.
Thankyou!
Justin Clark-Casey
OSVW Consulting
http://justincc.org
http://twitter.com/justincc
justincc@justincc.org
(please see next slides for references)
References
[Direct Service]
http://opensimulator.org/wiki/Direct_Service_Requests
[Graphing]
https://github.com/justincc/opensimulator-tools/tree/master/analysis/opensimulator-stats-analyzer
[iperf]
http://software.es.net/iperf/
[PCampbot]
http://opensimulator.org/wiki/PCampBot
References
[Performance]
http://opensimulator.org/wiki/Performance
[Siege]
http://www.joedog.org/siege-home/
[SRAS]
https://github.com/coyled/sras
[Stats]
http://opensimulator.org/wiki/Show_stats
[Texture Load Test]
https://github.com/justincc/opensimulator-tools/tree/master/test/texture-load-test
Q&A

Optimizing and Improving Opensimulator Performance

Uploaded by

Copyright:

Available Formats

Optimizing and Improving Opensimulator Performance

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Optimizing and Improving Opensimulator Performance

Uploaded by

Copyright:

Available Formats

Optimizing and Improving

The System Architecture

Components and interactions

Rules of thumb for good performance

Measuring and Testing

What the heck is going on?

400 connections for the keynote

Why listen to me?

OpenSimulator core developer and consultant for 7+ years.

ghosts branch to soon be integrated.

But even I'm not an expert on everything

Performance is a very complex topic in this architecture.

Many different network & server arrangements.

The System Architecture

Talking about a Hypergrid-enabled grid installation.

Though much also applies to standalone.

Much also applies to a non-Hypergrid installation.

Assume you are already familiar with fundamental

Standalone, grid, Hypergrid.

HTTP, TCP, UDP.

In general four aspects

Processing (CPU/GPU on viewer side)

for three components

Viewer Simulator, Services

UDP: need low latency

HTTP: need high bandwidth

Rule of thumb: try at least for 500 kbit per connection

Simulator Simulator same grid

HTTP: Low latency

Install Install via Hypergrid

HTTP: Low latency, good bandwidth

More prims, more avatars, higher load.

GPU/CPU limiting factor

More prims, more avatars, more scripts, higher load.

CPU limiting factor in the first instance

Recommended minimum 2 cores per simulator

One region vs multiple regions per simulator.

Asset storage dominates everything.

Deduplicating asset service in future release

All assets immutable so deletion extremely tricky.

Faster storage is better but probably not critical.

Allow as many assets to be cached as possible.

I know little about this.

Free threads for all! approach inefficient.

Improvements for conference.

Mono vs Windows .NET.

Offload GetTexture/GetMesh to direct service calls [DIRECT SERVICE]

Handling bad foreign installations in the Hypergrid.

Measuring and Testing

Every install can be different.

Very large number of moving parts

More operating system than app/webserver.

Identifying and reproducing an issue

Is critical to actually fix it.

Often much >50% of the work.

Generations of stat systems...

Dominant is now show stats all.

Console commands on simulator and services

Record raw numbers and moving averages