Shout_ml: Another Way To Block /yells

Langues: JP EN DE FR
users online
Forum » Windower » General » shout_ml: Another way to block /yells
shout_ml: Another way to block /yells
 Asura.Epigram
Offline
Serveur: Asura
Game: FFXI
user: jlisic
Posts: 109
By Asura.Epigram 2020-10-07 23:06:09
Link | Citer | R
 
Github shout_ml

Block Spam and Find Content


shout_ml started as a working experiment to see how effectively I can block spam using some simple machine learning. As opposed to other methods, it doesn't just classify spammy JP messages, instead it classifies each shout or yell into eight different categories. So you can select which shouts to see and which shouts to ignore.

Eight categories: Content, RMT, Merc, JP Merc, Chat, Selling (non-Merc), Buying (non-Merc), and Unknown.

To be more technical, this is a simple n-gram model using xgboost. All features currently used were trained back in February, so some things are slipping through, but the accuracy seems to be pretty good (around 95%). All scoring is done within the add-on, so it evaluates the 1,600 trees from the xgboost with every /yell. This is actually pretty fast and didn't seem to cause any noticeable lag for me.

A current version of the model has been provided with the add-on. I plan to start distributing updated models through my website once I start training more regularly. You will be able to get updates directly from the add-on once I add this functionality.

New Features as of version .81 (released October 14th):
  • 16 new features in the ml model.

  • Labeled 1,000 more shouts (Reading shouts for Asura for one hour makes me cry).

  • Updated model, overall accuracy went from .78 to .91.

  • Added a simple exception rule through "allow-lists".

  • Added command status, shows threshold values and allow-list

  • Added config saving

  • Did a lot of database work and cleaned up R scripts for building the model.

  • Addded content window!



I have created a demo video (basic usage, sorry about the audio):

YouTube Video Placeholder



Content window (helping you find things to do):
YouTube Video Placeholder
[+]
 Asura.Mims
Offline
Serveur: Asura
Game: FFXI
user: Ginza
Posts: 249
By Asura.Mims 2020-10-08 09:31:58
Link | Citer | R
 
This is really neat, I hope you keep working on this.

Here are some things I would really like to see in this addon:

Have one of the //sml commands display what the current blocking thresholds are set to. As it is I can only only see current threshold by actually changing to a new threshold.

Some kind of log export tool to track offensive shouts that get past the filter? As it is I'm seeing odd items get through that would seem obvious blocking targets, but I have no way to analyze them after the fact.

The first one is probably far easier than the second, I realize.
This addon has real promise, keep up the good work!

/edit:
Also, EXP merc seemed to get through a bit, maybe give it its own category or combine it with JP merc?
 Asura.Epigram
Offline
Serveur: Asura
Game: FFXI
user: jlisic
Posts: 109
By Asura.Epigram 2020-10-08 10:03:08
Link | Citer | R
 
Thanks, the exp merc I think is a newer shout on asura, so it never really made it into the training set. I think a bit more training would fix that. I am also planning on adding the rules from some of the rule based filters as methods.

Ideally, I would love to be able to do at some point is to be able to identify recent shouts for content /yells filtered by jobs I want to play on. I do a fair bit of pugging, so that's useful to me :)
 Asura.Mims
Offline
Serveur: Asura
Game: FFXI
user: Ginza
Posts: 249
By Asura.Mims 2020-10-08 10:11:52
Link | Citer | R
 
Ooh, whitelisting shouts based on the jobs they are looking for?
That is an excellent idea! Looking forward to future releases.
 Asura.Akaden
Offline
Serveur: Asura
Game: FFXI
By Asura.Akaden 2020-10-08 12:10:36
Link | Citer | R
 
This is great!

Some features I'd love to see to make this an extremely useful tool:
  • Saving settings. Thresholds, etc.

  • Exceptions for specific words. EG: "I don't really want to see all mercs, but I'd love to see if somebody's selling that one omen body I really need, or a pair of Regal Cuffs."

  • Toggling groups. This is pretty much as setting the threshold to 1, but I think a bit more obvious to people that don't know that.

  • Logging might help find outliers. Logging "near misses" for values close to the threshold might help too.

 Asura.Verbannt
Offline
Serveur: Asura
Game: FFXI
user: Akton
Posts: 164
By Asura.Verbannt 2020-10-08 15:51:41
Link | Citer | R
 
I would be satisfied if repeated shouts were just shown once per like 5 min. (*edit: or some amount of time we could set the interval)

I don't mind shouts for almost w/e reason, just the sheer volume of them. A party seeking a Geo / whm for a JP group being spammed is just as bad as the mercs IMO. Spam is Spam...

Additionally I know why people shout this often, as people zone into cities at varying times. I wish the game had a in game message board that something could be posted to and people had to go look for things on it compared to forming groups in shout.... Even though I form groups in shouts (since nothing of the sort exists)
[+]
 Asura.Epigram
Offline
Serveur: Asura
Game: FFXI
user: jlisic
Posts: 109
By Asura.Epigram 2020-10-09 13:58:05
Link | Citer | R
 
I've trained a new model, current overall accuracy is just shy of 90%, so I'll push this out tonight with new updates to the add-on.

For those interested, these are the estimates of the proportion of yells by category for Asura.
Code
Content   RMT  Merc  JP Merc  Chat  Selling Buying  Other
0.193   0.028 0.206  0.327    0.107 0.070   0.059   0.009 


This is also how the proportion has changed over time (or at least per the model). What we can see is that over time it looks like the proportion of chat has gone way down and JP merc/merc have greatly increased.




These are the current true positive rates and true negative rates by category. As you can see, buying and selling are a bit messy, this is mainly due to 'in pool' yells being a bit mixed in the training data. I either need to consider them a distinct category or push all of them to either Merc or Selling (I'm interested in any suggestions).
Code
  actual_label         tpr   tnr
  <chr>              <dbl> <dbl>
1 Content            0.848 0.964
2 RMT                1     1    
3 Merc               0.816 0.932
4 JP Merc            0.937 1    
5 Chat               0.96  0.953
6 Selling (non-Merc) 0.625 1    
7 Buying (non-Merc)  0.786 0.987
8 Unknown            0.333 1    


For the confusion matrix (I was too lazy to label it, just refer to the index on the prior tpr/tnr chart) you can see the mix up (between 3 and 6). Rows are observed, cols are predicted.
Code
    pred_1 pred_2 pred_3 pred_4 pred_5 pred_6 pred_7 pred_8
  1   0.16   0.00   0.02   0.00   0.00   0.00   0.00   0.00
  2   0.00   0.03   0.00   0.00   0.00   0.00   0.00   0.00
  3   0.01   0.00   0.17   0.02   0.00   0.02   0.01   0.00
  4   0.00   0.00   0.00   0.31   0.00   0.00   0.00   0.00
  5   0.02   0.00   0.01   0.00   0.10   0.00   0.00   0.01
  6   0.00   0.00   0.00   0.00   0.00   0.04   0.00   0.00
  7   0.00   0.00   0.01   0.00   0.00   0.00   0.05   0.00
  8   0.00   0.00   0.00   0.00   0.00   0.00   0.00   0.00
[+]
Offline
Posts: 330
By Lili 2020-10-09 17:31:27
Link | Citer | R
 
I absolutely love that you made this.

I don't have anything of value to say, except that JP mercs shot up right after the sparks nerf. Gotta earn that gil to resell somehow after all.
 Asura.Epigram
Offline
Serveur: Asura
Game: FFXI
user: jlisic
Posts: 109
By Asura.Epigram 2020-10-09 22:43:04
Link | Citer | R
 
Updated to version 0.6. Still need some tuning, I saw mrgrim's Ou shouts make it through so will be working on this.

I have created the basics needed to get what I need for a message box with last shouts for content by player. This will be updated in the next version. So much for playing tonight :)
Offline
Posts: 237
By Mrgrim 2020-10-09 23:19:57
Link | Citer | R
 
Asura.Epigram said: »
Updated to version 0.6. I saw mrgrim's Ou shouts make it through so will be working on this.


Ancient Elvaan Voodoo Magic!!!
[+]
 Asura.Akaden
Offline
Serveur: Asura
Game: FFXI
By Asura.Akaden 2020-10-10 11:45:45
Link | Citer | R
 
Sat in town for about an hour, not a single cp/jp/exp/aeonic shout. Compared with another person too, all of those were present in their shouts.
It's lovely.
 Asura.Epigram
Offline
Serveur: Asura
Game: FFXI
user: jlisic
Posts: 109
By Asura.Epigram 2020-10-10 21:21:39
Link | Citer | R
 
Pushed out another update. Saving allow words now works.

Next major update will have windows for last shouts and time since last shout.
[+]
 Asura.Akaden
Offline
Serveur: Asura
Game: FFXI
By Asura.Akaden 2020-10-11 10:33:52
Link | Citer | R
 
Asura.Epigram said: »
Pushed out another update. Saving allow words now works.

Next major update will have windows for last shouts and time since last shout.
Just tried the allow words, love it.

If I may suggest a strategy for saving using windower's settings them better? It should simplify it and allow toggling words without having to use an index.

Save it as a comma-separated string (all lowercase) and then on load, break it into a set. Sets are great because they can only ever contain one of any word, and they're super fast to check if a word is inside.
If you want I can send a pull request with the change. It won't be much trouble.
I submitted a PR :)
 Asura.Epigram
Offline
Serveur: Asura
Game: FFXI
user: jlisic
Posts: 109
By Asura.Epigram 2020-10-11 20:35:09
Link | Citer | R
 
Accepted pull request, added new model with new features. Combined accuracy is at 91%.

As a todo: sets seem to be working, but status isn't. I'll work this through sometime this week. done!

Also fixed a weird issue when reading from the leaf nodes of the tree where the boosted value = -0.
[+]
 Asura.Epigram
Offline
Serveur: Asura
Game: FFXI
user: jlisic
Posts: 109
By Asura.Epigram 2020-10-12 21:49:24
Link | Citer | R
 
I still need to implement hiding the window, clearing rows, and filters, but I now have a working current content window.

As a note, this is the unreleased version. When this feature is ready I'll push it out :)

YouTube Video Placeholder
 Asura.Epigram
Offline
Serveur: Asura
Game: FFXI
user: jlisic
Posts: 109
By Asura.Epigram 2020-10-14 23:35:52
Link | Citer | R
 
*** Bump *** Version update with a content window and 95% accuracy hit for classification!

The content window provides all current content related shouts.

Shouts for Ambuscade (non-merc) are there!
Shouts for Omen (non-merc) are there!
Shouts for Dynamis D (non-merc) are there!
Shouts for Odyssey (I've never seen one) but they probably will be there!
Shouts for anything that is classified as content will end up there. (hot-tub time machine RDM, relic BLM 4/6 "do you need it?") would probably show up.
Shouts for your mom! (not going to judge) they might be there!

Note that not everyone has a shout blocker, so some people do look a bit spammy. But it's not them, it's just all the garbage being shouted.

YouTube Video Placeholder
[+]
 Asura.Epigram
Offline
Serveur: Asura
Game: FFXI
user: jlisic
Posts: 109
By Asura.Epigram 2020-10-24 17:20:03
Link | Citer | R
 
shout_ml 1.0 released.

Over 95% accuracy!
Fixed issue where the currency feature was not be applied correctly.
Removed the blocking in the content window for now (will rework with a later version).
Shouts now wrap to a new line instead of being truncated in the content window.
Alpha Channel set to 155 for default in the content window (partial translucency).

YouTube Video Placeholder
 Asura.Sadaharu
Offline
Serveur: Asura
Game: FFXI
user: Sadaharu
Posts: 2
By Asura.Sadaharu 2020-11-14 00:49:18
Link | Citer | R
 
This is awesome, thank you!
 Asura.Suteru
Offline
Serveur: Asura
Game: FFXI
user: Suteru
Posts: 484
By Asura.Suteru 2020-11-14 04:00:48
Link | Citer | R
 
Is there some way to tell this plugin that a shout was indeed a certain type? It works great but some very obvious ones still go through ("500/4m" etc JP mercs, etc)
Offline
Serveur: Asura
Game: FFXI
Posts: 36
By Asura.Jugsofholyness 2020-12-06 02:05:04
Link | Citer | R
 
How do I auto load this addon?
 Asura.Arico
Offline
Serveur: Asura
Game: FFXI
user: Tename
Posts: 494
By Asura.Arico 2020-12-06 02:22:56
Link | Citer | R
 
Asura.Jugsofholyness said: »
How do I auto load this addon?

//lua l shout_ml
necroskull Necro Bump Detected! [53 days between previous and next post]
 Asura.Epigram
Offline
Serveur: Asura
Game: FFXI
user: jlisic
Posts: 109
By Asura.Epigram 2021-01-28 12:52:09
Link | Citer | R
 
Just added an experimental feature to get messages while running around in non-town zones.

shout_ml on github

To turn it on just type (off by default):
Code
//sml p


This is a community driven approach and relies on people on your server running shout_ml with logging enabled (off by default). To help contribute your shouts to people running outside of /yell range type:
Code
//sml l


Your shouts will be used to help score future models and I only record the following three items: person shouting, message, and server (Asura etc...). I don't care or want anything else.

Added Video:
YouTube Video Placeholder
 Asura.Epigram
Offline
Serveur: Asura
Game: FFXI
user: jlisic
Posts: 109
By Asura.Epigram 2021-01-29 21:27:02
Link | Citer | R
 
Released Version 1.5.1:

  • Fixed issue with JP merc. shouts causing an runtime (non-fatal) error

  • Fixed right alignment of text issue when getting out-of-town shouts/yells

  • Created a web page that shows current shouts on each server when someone is logging (sorry I only know the Asura server id (31), if someone has a resource for the others, please let me know)




Next version will be 2.0 and pretty much feature complete, this will include automatic model updates.
necroskull Necro Bump Detected! [65 days between previous and next post]
Offline
By Agrestic 2021-04-04 10:46:23
Link | Citer | R
 
Did this lose effectiveness recently? A growing number of merc shouts are spilling through it nowadays.
 Asura.Wotasu
Online
Serveur: Asura
Game: FFXI
user: Wotasu
Posts: 195
By Asura.Wotasu 2021-04-04 11:36:09
Link | Citer | R
 
They've just altered their yells abit so it slips through sadly. So might need an update of some sort.
[+]
necroskull Necro Bump Detected! [85 days between previous and next post]
 Phoenix.Godofwars
Offline
Serveur: Phoenix
Game: FFXI
Posts: 2
By Phoenix.Godofwars 2021-06-28 15:13:18
Link | Citer | R
 
how do i just block the EXP/JP/CP ppl merc only i dont mind the other merc that sell stuff or mercs endgame nm's but them are blocked aswell thanks
Log in to post.