Renegade Public Forums
C&C: Renegade --> Dying since 2003™, resurrected in 2024!
Home » Tiberian Technologies / Blackhand Studios » Tiberian Technologies Forum » Introduction to the SSGM 4.0 TCP Logging System
Introduction to the SSGM 4.0 TCP Logging System [message #434694] Fri, 13 August 2010 06:49 Go to next message
danpaul88 is currently offline  danpaul88
Messages: 5795
Registered: June 2004
Location: England
Karma: 0
General (5 Stars)
This topic is intended for bot developers who are interested in supporting SSGM 4.0, which will be released as part of the scripts.dll 4.0 project.

Introduction

The file based logging system under previous versions of SSGM is no longer supported and bots which rely on the ability to read ssgmlog and gamelog files will no longer function correctly. However the renlog files will still be created to allow some limited compatibility with older bots.

Instead, SSGM 4.0 supports a new type of logging which server owners based on a TCP connection between the bot and the server. Once connected to the server bots will receive a steady stream of log messages through the connection.

In order to ease the process of updating bots to the new SSGM logging system it was decided to implement the existing ssgmlog and gamelog messages into the new system in the same format as they used in the old file based logging system. This means that bots can simply point messages to existing parsing procedures without having to significantly rewrite their log handling code.




Protocol

The logging protocol is extremely simple. The first three characters if each message sent through the protocol are digits between 0 and 9 which indicate the message type. The remainder of the message is the log entry. Each message is terminated with the NULL character (0x0).

There are 4 built in message types;
000 - SSGM log messages
001 - Gamelog messages
002 - Renlog messages (a copy of everything written to the renlog files)
003 - Console Output (a copy of everything written to the console, basically a copy of 002 without timestamps)

Here are some examples of messages a bot might receive through the TCP connection;

001[14:06:20] POS;VEHICLE;1500000126;GDI_Ceiling_Gun_AGT;-66;21;9;0;100;100

001[14:06:21] POS;SOLDIER;1500000206;CnC_GDI_MiniGunner_0;-12;27;0;179;100;100

003Total current bandwidth usage for players is 85 kilobits per second

002[14:06:22] Total current bandwidth usage for players is 85 kilobits per second

003Westwood Online mode active since 11/08/2010 - 19:06:54

002[19:18:46] Westwood Online mode active since 11/08/2010 - 19:06:54



Important Note
For reasons which are too complicated to go into at this point the first line of the player_info response (the column headers) is output with a newline character (0xD) at the front of the line. If your bot needs to parse this line and would choke on that character you should use an appropriate function to strip that character out of any messages before forwarding them for parsing.




Configuration
SSGM 4.0 uses a configuration file named ssgm.ini, located in the data folder of the FDS installation. There are two important configuration parameters in this file for the logging system.

[General]
Port=8080
EnableGamelog=true


The port parameter defines the port number on which the server should listen for incoming TCP connections to the logging system and should be different from the RenRem and game ports defined in server.ini. Bots must read the port number from this file (or have it provided to them via config settings) to allow them to connect to the server.

The EnableGamelog option turns the logging system on or off and must be set to true to enable the logging system.




Testing

Right now your probably thinking something along the lines of "yeah, this is great and all, but how can I support something I can't test?". Don't worry, I thought of that problem in advance and have prepared a test server which you may connect your bots to in order to test their handling of the protocol.

Server: 82.11.23.160
TCP Log Port: 37854

You can interact with the test server by joining it (lmsbcdan in WOL or 82.11.23.160:4848 by direct connect) and by joining the IRC channel #tt_ssgm_bottest on irc.n00bstories.com where BRenBot 1.53 is running to help you test your bots responses to the TCP logging information.

Please be aware that this server is for bot testing purposes only and should not be used for normal gameplay... apart from anything else it will probably lag due to being hosted on a fairly slow connection.


Server is now offline due to low demand, if you need the server running again to test your bot(s) please send me a PM and I will load it up again.




More Information / Help

If you need any more information or need help with anything related to this new logging system please reply here or send me a PM on these forums.

Have fun testing!


http://steamsignature.com/card/1/76561197975867233.png

[Updated on: Sun, 26 September 2010 02:12]

Report message to a moderator

Re: Introduction to the SSGM 4.0 TCP Logging System [message #434696 is a reply to message #434694] Fri, 13 August 2010 06:52 Go to previous messageGo to next message
CarrierII is currently offline  CarrierII
Messages: 3804
Registered: February 2006
Location: England
Karma: 0
General (3 Stars)

Awesome.

As a technical note to non-techies who are interested in the probable progress of TT, the fact that a protocol has been publicly documented suggests they are very far along.

Smile


Renguard is a wonderful initiative
Toggle Spoiler
Re: Introduction to the SSGM 4.0 TCP Logging System [message #434700 is a reply to message #434694] Fri, 13 August 2010 07:35 Go to previous messageGo to next message
reborn is currently offline  reborn
Messages: 3231
Registered: September 2004
Location: uk - london
Karma: 0
General (3 Stars)
It's great that disk based logging is being dropped.
It's cool that you decided to use TCP so people can host their bots on a different server if need be. I tend to think that won't be used, and you might of saved yourself some time using named pipes, maybe? Ofcourse, it's nice to have the option to ship the data offsite though. Smile

Does it send the data in raw format or does it use bit packing or something?



Re: Introduction to the SSGM 4.0 TCP Logging System [message #434701 is a reply to message #434694] Fri, 13 August 2010 07:42 Go to previous messageGo to next message
danpaul88 is currently offline  danpaul88
Messages: 5795
Registered: June 2004
Location: England
Karma: 0
General (5 Stars)
It's just sent as ASCII over the network, as shown in the protocol outlined above. Crude but effective.

As for using TCP, because it allows multiple simultaneous connections it allows you to do some useful stuff like off-server rank calculations on a separate bot dedicated to the task, where it doesn't matter if it lags behind real time as it doesn't need to respond to user input. At busy times it could buffer input and process it as fast as it can, and catch up to real time during an idle period.

Lots of possibilities...


http://steamsignature.com/card/1/76561197975867233.png

[Updated on: Fri, 13 August 2010 07:43]

Report message to a moderator

Re: Introduction to the SSGM 4.0 TCP Logging System [message #434702 is a reply to message #434694] Fri, 13 August 2010 07:47 Go to previous messageGo to next message
reborn is currently offline  reborn
Messages: 3231
Registered: September 2004
Location: uk - london
Karma: 0
General (3 Stars)
Ofcourse, it's a nice option to have. I fear though that there are few likely to take advantage of it.

However, if it is going to be sent off-site, it may prove a good idea to pack the data being sent to cut down on network traffic and load? Just an idea.. I think it's great that disk based dropping has been dropped, anything over that is just a big bonus. Very Happy

The built in logging of renlog and console output, have you hooked these functions and stopped them being written to disk, and instead being sent via the TCP connection, or are they still being written to disk, too?



Re: Introduction to the SSGM 4.0 TCP Logging System [message #434703 is a reply to message #434702] Fri, 13 August 2010 07:58 Go to previous messageGo to next message
danpaul88 is currently offline  danpaul88
Messages: 5795
Registered: June 2004
Location: England
Karma: 0
General (5 Stars)
reborn wrote on Fri, 13 August 2010 15:47

The built in logging of renlog and console output, have you hooked these functions and stopped them being written to disk, and instead being sent via the TCP connection, or are they still being written to disk, too?



danpaul88 wrote on Fri, 13 August 2010 14:49

However the renlog files will still be created to allow some limited compatibility with older bots.




Wink

Some still *some* disk logging, but nowhere near as much.


http://steamsignature.com/card/1/76561197975867233.png

[Updated on: Fri, 13 August 2010 07:58]

Report message to a moderator

Re: Introduction to the SSGM 4.0 TCP Logging System [message #434704 is a reply to message #434694] Fri, 13 August 2010 08:00 Go to previous messageGo to next message
halo2pac is currently offline  halo2pac
Messages: 659
Registered: December 2006
Location: Near Cleveland, Ohio
Karma: 0
Colonel
You can download a beta of this already if you have been following my projects.

Ive had this done for some time now and supports creation of objects and internal commands.

and it works with the current ssgm.

http://halo2pac.99k.org/fdscomm/betas/FDSComm%20v0.4b.zip
http://halo2pac.99k.org/fdscomm/guide.htm

Its a plugin so I didn't include source.


http://img339.imageshack.us/img339/1991/nefobbygenyunoreleasere.jpg
Rene-Buddy | Renegade X
Join the fight against Obsessive-Compulsive Posting Disorder. Cancel is ur friend.
*Renegade X Dev Team Member*

[Updated on: Fri, 13 August 2010 08:02]

Report message to a moderator

Re: Introduction to the SSGM 4.0 TCP Logging System [message #434705 is a reply to message #434694] Fri, 13 August 2010 08:02 Go to previous messageGo to next message
reborn is currently offline  reborn
Messages: 3231
Registered: September 2004
Location: uk - london
Karma: 0
General (3 Stars)
Sorry, I over-looked that. it's so rare that something awesome happens around renegade that I read it in a rush.

How does that work then? Does it read the disk file and send it off, or does it intercept that information before it's written to renlog?

It's great to see some news come out of this forum, well done Dan Paul.



Re: Introduction to the SSGM 4.0 TCP Logging System [message #434706 is a reply to message #434694] Fri, 13 August 2010 08:22 Go to previous messageGo to next message
Hitman is currently offline  Hitman
Messages: 878
Registered: November 2005
Location: Belgium
Karma: 0
Colonel
and what does all that mean in a language people can understand?
Re: Introduction to the SSGM 4.0 TCP Logging System [message #434710 is a reply to message #434694] Fri, 13 August 2010 08:31 Go to previous messageGo to next message
reborn is currently offline  reborn
Messages: 3231
Registered: September 2004
Location: uk - london
Karma: 0
General (3 Stars)
The servers you play in log all the events and shit that happens in-game to the hard drives, this is called disk input-output. It's a bottle neck and is CPU intensive, so a server's SFPS can drop due to this (OMG LAG!).

Not logging to disk stops that. Meaning less server lag and the ability to increase the max player limit.



Re: Introduction to the SSGM 4.0 TCP Logging System [message #434712 is a reply to message #434694] Fri, 13 August 2010 08:38 Go to previous messageGo to next message
Omar007 is currently offline  Omar007
Messages: 1711
Registered: December 2007
Location: Amsterdam
Karma: 0
General (1 Star)
This is pretty awesome guys Thumbs Up

Though just wondering; why not make a bot (embedded) in scripts.dll?? Razz


http://tiberiumredux.omarpakker.nl/Old Unused Parts/Plaatjes/PromoteBanner_Hades_small.jpg

[Updated on: Fri, 13 August 2010 08:38]

Report message to a moderator

Re: Introduction to the SSGM 4.0 TCP Logging System [message #434714 is a reply to message #434710] Fri, 13 August 2010 08:45 Go to previous messageGo to next message
Starbuzzz
Messages: 1637
Registered: June 2008
Karma: 0
General (1 Star)
Hitman wrote on Fri, 13 August 2010 10:22

and what does all that mean in a language people can understand?


thanks for the question!

reborn wrote on Fri, 13 August 2010 10:31

The servers you play in log all the events and shit that happens in-game to the hard drives, this is called disk input-output. It's a bottle neck and is CPU intensive, so a server's SFPS can drop due to this (OMG LAG!).

Not logging to disk stops that. Meaning less server lag and the ability to increase the max player limit.


and thanks for the answer!

edit:

and thanks for the update danpaul!

[Updated on: Fri, 13 August 2010 08:45]

Report message to a moderator

Re: Introduction to the SSGM 4.0 TCP Logging System [message #434715 is a reply to message #434694] Fri, 13 August 2010 08:47 Go to previous messageGo to next message
halo2pac is currently offline  halo2pac
Messages: 659
Registered: December 2006
Location: Near Cleveland, Ohio
Karma: 0
Colonel
Omar007 wrote on Fri, 13 August 2010 10:38

This is pretty awesome guys Thumbs Up

Though just wondering; why not make a bot (embedded) in scripts.dll?? Razz

Its called Yarr, I believe and Daniel made it.


Some people (including me) want this functionality because its just awesome to create your own bot. Especially if you know how.

I know a million ways to make a bot and I have but its private, because I barely work on since I have a life and I program for a living. That and I'm getting married soon.

Also [HB] is too complicated since it is like 6 bots in one so it has to be run a quad core. I added text messaging commands but that was a waste of time. that and TS voice commands.


http://img339.imageshack.us/img339/1991/nefobbygenyunoreleasere.jpg
Rene-Buddy | Renegade X
Join the fight against Obsessive-Compulsive Posting Disorder. Cancel is ur friend.
*Renegade X Dev Team Member*

[Updated on: Sun, 15 August 2010 15:24]

Report message to a moderator

Re: Introduction to the SSGM 4.0 TCP Logging System [message #434730 is a reply to message #434710] Fri, 13 August 2010 11:21 Go to previous messageGo to next message
ExEric3 is currently offline  ExEric3
Messages: 746
Registered: February 2005
Location: Slovakia
Karma: 0
Colonel
reborn wrote on Fri, 13 August 2010 17:31

The servers you play in log all the events and shit that happens in-game to the hard drives, this is called disk input-output. It's a bottle neck and is CPU intensive, so a server's SFPS can drop due to this (OMG LAG!).

Not logging to disk stops that. Meaning less server lag and the ability to increase the max player limit.


But if you put RenFDS to ramdrive its much better. Big Ups

About what reborn talking here you can see on n00bstories server and how SFPS drops when someone join.

TCP Logging system is nice. I think in current SSGM we have RenLog Monitor which doesnt works at least for me and its over UDP. But on this (RenLog Monitor) working Dragonade Mode with DragonServ.
Re: Introduction to the SSGM 4.0 TCP Logging System [message #434733 is a reply to message #434705] Fri, 13 August 2010 13:14 Go to previous messageGo to next message
danpaul88 is currently offline  danpaul88
Messages: 5795
Registered: June 2004
Location: England
Karma: 0
General (5 Stars)
reborn wrote on Fri, 13 August 2010 16:02

It's great to see some news come out of this forum, well done Dan Paul.


Just to clarify, whilst I posted the documentation and provided a bit of help with the network stuff I did not do the bulk of the work on the TCP logging, you have jonwil to thank for that (I would have done it had I not been so busy with RL stuff).

Don't want anyone to think I am claiming credit for work I didn't do Smile


http://steamsignature.com/card/1/76561197975867233.png
Re: Introduction to the SSGM 4.0 TCP Logging System [message #434734 is a reply to message #434694] Fri, 13 August 2010 13:20 Go to previous messageGo to next message
cAmpa is currently offline  cAmpa
Messages: 597
Registered: March 2006
Karma: 0
Colonel
Great would be a beta for testing. Tell Me

Bückstabü!
Re: Introduction to the SSGM 4.0 TCP Logging System [message #434772 is a reply to message #434694] Sat, 14 August 2010 10:26 Go to previous messageGo to next message
trooprm02 is currently offline  trooprm02
Messages: 3266
Registered: August 2005
Location: Canada
Karma: 0
General (3 Stars)
When will the actually SSGM 4.0 server be going up and who will be hosting it?

Re: Introduction to the SSGM 4.0 TCP Logging System [message #434796 is a reply to message #434694] Sat, 14 August 2010 22:44 Go to previous messageGo to next message
jonwil is currently offline  jonwil
Messages: 3557
Registered: February 2003
Karma: 0
General (3 Stars)

I dont know when the actual proper SSGM 4.0 testing server (the one normal players are supposed to join and play on) will be live, we want to make sure its as bug free as we can get it first.

I believe BlackIntel will be hosting it on one of their boxes.


Jonathan Wilson aka Jonwil
Creator and Lead Coder of the Custom scripts.dll
Renegade Engine Guru
Creator and Lead Coder of TT.DLL
Official member of Tiberian Technologies
Re: Introduction to the SSGM 4.0 TCP Logging System [message #434917 is a reply to message #434694] Tue, 17 August 2010 06:32 Go to previous messageGo to next message
dr3w2 is currently offline  dr3w2
Messages: 485
Registered: September 2006
Location: Ottawa,Canada
Karma: 0
Commander
This is awesome. Nicely done.

n00bstories Server Administrator

[Updated on: Tue, 17 August 2010 06:32]

Report message to a moderator

Re: Introduction to the SSGM 4.0 TCP Logging System [message #434942 is a reply to message #434694] Tue, 17 August 2010 17:20 Go to previous messageGo to next message
Ethenal is currently offline  Ethenal
Messages: 2532
Registered: January 2007
Location: US of A
Karma: 0
General (2 Stars)

So if I'm reading correctly, some logs will still be written to disk (renlog), but now the SSGM and gamelogs? If so, I understand why, but that'll break all the older bots and I don't foresee any up-to-date replacements appearing. However, I imagine it wouldn't take too long to add the code for logging it to disk again so I suppose that doesn't really matter.

Either way, great job! Some places have used a system like this before and it has actually made a significant difference. So hopefully we'll see some improvements in lag. Smile

P.S. Is it just me who cringes at halo's posts?


-TLS-DJ-EYE-K wrote on Mon, 18 March 2013 07:29

Instead of showing us that u aren't more inteligent than a Toast, maybe you should start becomming good in renegade Thumbs Up

Re: Introduction to the SSGM 4.0 TCP Logging System [message #434953 is a reply to message #434942] Tue, 17 August 2010 20:50 Go to previous messageGo to next message
trooprm02 is currently offline  trooprm02
Messages: 3266
Registered: August 2005
Location: Canada
Karma: 0
General (3 Stars)
Ethenal wrote on Tue, 17 August 2010 19:20

I don't foresee any up-to-date replacements appearing.


Danpaul is currently writing up a new, matching BR actually.


[Updated on: Tue, 17 August 2010 20:50]

Report message to a moderator

Re: Introduction to the SSGM 4.0 TCP Logging System [message #434958 is a reply to message #434942] Wed, 18 August 2010 00:28 Go to previous messageGo to next message
reborn is currently offline  reborn
Messages: 3231
Registered: September 2004
Location: uk - london
Karma: 0
General (3 Stars)
Ethenal wrote on Tue, 17 August 2010 20:20

So if I'm reading correctly, some logs will still be written to disk (renlog), but now the SSGM and gamelogs? If so, I understand why, but that'll break all the older bots and I don't foresee any up-to-date replacements appearing. However, I imagine it wouldn't take too long to add the code for logging it to disk again so I suppose that doesn't really matter.

Either way, great job! Some places have used a system like this before and it has actually made a significant difference. So hopefully we'll see some improvements in lag. Smile

P.S. Is it just me who cringes at halo's posts?


I did this for the APB server and thought it would take Oxi ages to re-write the bot to manage this new method. He was done in less than an hour. I do not know which languages other bots are written in, but PERL is extremely easy to open up and start receiving TCP connections (or infact any socket).

The main bulk of functions in the renegade bots are to process the data they are reading, the functions to read the data make up very little of the bot itself.



Re: Introduction to the SSGM 4.0 TCP Logging System [message #434969 is a reply to message #434694] Wed, 18 August 2010 03:13 Go to previous messageGo to next message
StealthEye is currently offline  StealthEye
Messages: 2518
Registered: May 2006
Location: The Netherlands
Karma: 0
General (2 Stars)

It isn't a huge difference whether you are reading from a file or from a socket. The only significant difference is that a socket may disconnect at any time, which shouldn't lead to crashes or unwanted behavior. Otherwise, it's most just find&replacing a few things and that's it; especially if the application was well-designed, it should be an easy change.

Anyway, you can always make a new bot that simply listens to the socket and echos the data to a file. It's not preferred, but it is a possibility for old bots for which the source is unavailable.


BlackIntel admin/founder/coder
Please visit http://www.blackintel.org/

[Updated on: Wed, 18 August 2010 03:14]

Report message to a moderator

Re: Introduction to the SSGM 4.0 TCP Logging System [message #434971 is a reply to message #434694] Wed, 18 August 2010 03:21 Go to previous messageGo to next message
danpaul88 is currently offline  danpaul88
Messages: 5795
Registered: June 2004
Location: England
Karma: 0
General (5 Stars)
When I created this topic BRenBot had already been updated to fully support the new logging system, that's what is running on the test server for the SSGM 4.0 TCP logging system (see first post). I know for a fact that nightma12 has seen this topic (I linked him to it) and I believe he will be updating NR to support it to.

Due to the log messages being exactly the same, just sent via a socket instead of the disk, it should be relatively simple to update bots to connect to a socket, check the first three digits for the message type and pass it to their existing code to handle the various message types.


http://steamsignature.com/card/1/76561197975867233.png

[Updated on: Wed, 18 August 2010 03:22]

Report message to a moderator

Re: Introduction to the SSGM 4.0 TCP Logging System [message #437021 is a reply to message #434694] Sun, 26 September 2010 02:13 Go to previous messageGo to previous message
danpaul88 is currently offline  danpaul88
Messages: 5795
Registered: June 2004
Location: England
Karma: 0
General (5 Stars)
I have taken the testing server offline for the time being due to low demand, if any of you want to test your bot(s) at any time please send me a PM and I can load it up again.

http://steamsignature.com/card/1/76561197975867233.png
Previous Topic: January 1 - 2010
Next Topic: Few questions and possible update list?
Goto Forum:
  


Current Time: Mon Oct 14 01:47:11 MST 2024

Total time taken to generate the page: 0.01584 seconds