{"id":22447,"date":"2020-11-17T10:41:37","date_gmt":"2020-11-17T10:41:37","guid":{"rendered":"https:\/\/www.experfy.com\/blog\/user-agent-strings-parsing-ml-models\/"},"modified":"2021-11-25T09:12:06","modified_gmt":"2021-11-25T09:12:06","slug":"user-agent-strings-parsing-ml-models","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/ai-ml\/user-agent-strings-parsing-ml-models\/","title":{"rendered":"Still Parsing User-Agent Strings for Your Machine Learning Models?"},"content":{"rendered":"\n

Use This Instead<\/em><\/p>\n\n\n\n

Information contained in User-Agent strings can be efficiently represented using low-dimensional embeddings, and then employed in downstream Machine Learning tasks.<\/p><\/blockquote>\n\n\n\n

What on Earth are User-Agent strings?<\/h2>\n\n\n\n

When a user interacts with a website, the browser sends HTTP requests to the server to fetch the required content, submit data, or perform other actions. Such requests typically contain several headers<\/a>, i.e. character key-value pairs that specify parameters of a given request. A User-Agent string<\/a> (refered to as \u201cUAS\u201d below) is an HTTP request header that describes the software acting on the user\u2019s behalf (Figure 1).<\/p>\n\n\n\n

\"Still
Figure 1. The browser acts as the user\u2019s agent when sending requests to the server. The User-Agent string describes properties of the browser.<\/figcaption><\/figure><\/div>\n\n\n\n

The original purpose of UAS was content negotiation<\/em><\/a>, i.e. a mechanism of determining the best content to serve to a user depending on the information contained in the respective UAS (e.g., image format, the language of a document, text encoding, etc.). This information typically includes details on the environment the browser runs in (device, operating system and its version, locale), browser engine and version, layout engine and version, and so on (Figure 2).<\/p>\n\n\n\n

\"Example
Figure 2. Example of a typical UAS and its elements.<\/figcaption><\/figure><\/div>\n\n\n\n

Although serving different web pages to different browsers is considered a bad idea<\/a> nowadays, UAS still have many practical applications. The most common one is web analytics<\/em>, i.e. reporting on the traffic composition to optimise the effectiveness of a website. Another use case is web traffic management<\/em>, which involves blocking nuisance crawlers, decreasing the load on a website from unwanted visitors, preventing click fraud<\/a>, and other similar tasks. Due to the rich information they contain, UAS can also serve as a source of data for Machine Learning applications<\/em>. However, the latter use case has not received much attention so far. Here I address this issue and discuss an efficient way to create informative features from UAS for Machine Learning models. This article is a summary of the talk I presented recently at two conferences \u2014 Why R?<\/em><\/a> and Enterprise Applications of the R Language<\/em><\/a>. Data and code used in the examples described below are available at GitHub<\/a>.<\/p>\n\n\n\n

UAS elements as features for Machine Learning models<\/h2>\n\n\n\n

UAS elements can often serve as useful proxies of user characteristics, such as lifestyle, tech-savviness, and even affluence. For example, a user, who is typically visiting a website from a high-end mobile device is likely different from a user visiting that same website from Internet Explorer on a desktop computer running Windows XP. Having UAS-based proxies for such characteristics can be particularly valuable when no other demographic information is available for a user (e.g. when a new, unidentified person visits a website).<\/p>\n\n\n\n

In certain applications, it can also be useful to distinguish between human and non-human web traffic. In some cases, this is straightforward to do as automated web crawlers use a simplified UAS format that includes the word \u201cbot\u201d (e.g., Googlebot\/2.1 (+http:\/\/www.google.com\/bot.html)<\/code>). However, some crawlers do not follow this convention (e.g., Facebook bots contain<\/a> the facebookexternalhit<\/code> word in their UAS), and identifying them requires a lookup dictionary.<\/p>\n\n\n\n

One seemingly straightforward way to create Machine Learning features from UAS is to apply a parser<\/em>, extract individual UAS elements, and then one-hot encode these elements. This approach can work well in simple cases when only high-level and readily identifiable UAS elements need transforming into features. For example, it is relatively easy to determine the type of hardware of a User Agent (mobile vs desktop vs server, etc.). Several high-quality, regular expressions-based parsers can be used for this kind of feature engineering (e.g., see the \u201cua-parser\u201d project<\/a> and its implementations for a selection of languages).<\/p>\n\n\n\n

However, the above approach quickly becomes impractical when one wants to use all<\/em> elements<\/em> making up a UAS and extract maximum information from them. There are two main reasons for this:<\/p>\n\n\n\n