There has been an increase of Robot traffic over the last 12 months and we have been working hard at distinguishing between legitimate and illegitimate traffic. In Financial Services it is imperative to provide value for every single click and we do not want to be wasting our clients’ budgets on artificially generated traffic that has no chance of ever becoming a customer. dianomi has always been about long term mutually beneficial working relationships and trust, so it is key to keep on top of this situation.
There are standard ways of robots identifying themselves, and for us to tell robots where they should and should not click (“robots.txt”). Crawlers from the major search engines will follow these rules and therefore don’t present a problem. Robots that don’t obey these rules can still usually be identified by behaving in a way that is evidently not human. However we are finding some robots are becoming more sophisticated in looking like a human – by for example having a User-Agent which is usually used by a browser, supporting javascript and spreading the clicks over a longer time frame and over several IP addresses.
Back in 2012 we were voiding only 11% of clicks, but we have seen a big increase in robot traffic since, and particularly this year resulting in us voiding a significant higher percentage of clicks. It is certainly a challenge but we are continually improving our internal metrics to highlight suspicious activity and run daily checks on our global click traffic. We will also void clicks retrospectively if we see a pattern which looks non-human.
We are giving our clients full transparency of their clicks with time, ip address, user agent and other data, and whether we have validated or voided them. With this greater transparency and our continuous work at identifying robots our Financial clients can be confident in the quality of the clicks they are being charged for, and if they have any suspicions they can review the data for themselves and get back to us with their concerns.
Transparency and trust build long lasting relationships.
Michael Smith, Head of IT
