{"id":9209,"date":"2020-08-04T08:01:25","date_gmt":"2020-08-04T08:01:25","guid":{"rendered":"https:\/\/www.experfy.com\/blog\/?p=9209"},"modified":"2023-11-23T10:49:36","modified_gmt":"2023-11-23T10:49:36","slug":"getting-your-machine-learning-model-out-to-the-real-world","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/ai-ml\/getting-your-machine-learning-model-out-to-the-real-world\/","title":{"rendered":"Getting Your Machine Learning Model Out to the Real World"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n

Anexciting development in the past few years is the proliferation of machine learning in the product. We now see state-of-the-art computer vision models being deployed on mobile phones. State-of-the-art natural language processing models are being used to\u00a0improve<\/a>\u00a0search<\/a>.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Ut elit tellus, luctus nec ullamcorper mattis, pulvinar dapibus leo.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

While there have been many articles on new, exciting machine learning algorithms<\/a>, there isn\u2019t as many on productizing machine learning models. There has been upcoming interest in the engineering of these systems, such as the\u00a0ScaledML conference, and the birth of\u00a0MLOps<\/a>. However, productionizing machine learning models is a skill that is still in short supply.<\/p>\n\n\n\n

In my career so far, I\u2019ve gotten a few hard-earned lessons in productionizing machine learning models. Here are some of them.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

Build or Buy?<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Twitter recently has been all abuzz about\u00a0GPT-3<\/a>. Primed with the right\u00a0seed<\/em>, it turns out that GPT-3 can generate do many amazing things such as\u00a0generating buttons on webpages<\/a>\u00a0and\u00a0React code<\/a>. Spurred by these exciting developments, your stakeholder has an idea: what if you build a GPT-3 equivalent to help speed up the development of the frontend of the company you work at using GPT-3?<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Should you train your own GPT-3 model?<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Well, no! GPT-3 is a 175 billion parameter model that requires the resources most companies outside of the likes of Google and Microsoft would be hard-pressed to develop. Since OpenAI is trialling a beta for its GPT-3 API, it would be more prudent to try it out, and see if the model solves your use case, and the API scales to your use case.<\/p>\n\n\n\n

So then, to buy (more like rent) or build? It depends.<\/p>\n\n\n\n

It may look like a cop-out answer, but it is an age-old question in the software industry. It is also such a large and complicated topic, that it deserves a post of its own.<\/p>\n\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n

However, here I can offer some guidelines that can help you discuss buying vs. building question with a stakeholder. The first is<\/p>\n\n\n\n

\n

Is this use case not a core competence? Is it generic enough for an existing\/potential AWS\u00b9 offering?<\/p>\n<\/blockquote>\n\n\n\n

The underlying principle of this question is to ask yourself if\u00a0what you\u2019re about to work on is part of the company\u2019s core competence<\/strong>. If your stakeholders can afford it, it may be worth buying an off-the-shelf solution.<\/p>\n\n\n\n

\n

Are resources besides data available? E.g., talent and infrastructure?<\/p>\n<\/blockquote>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

At its heart, building and deploying a reliable machine learning system is\u00a0still mostly software engineering, with an added twist:\u00a0data management<\/em><\/strong>. If the team doesn\u2019t have experience building models as service APIs, with collecting, cleaning, labelling and storing data, then the amount of time needed can be prohibitive, say about 1 to 2 years (or even more). Instead, deploying a third party solution could take 3 to 6 months since it\u2019s mostly about integrating the solution into the product. Business owners would find this savings of time and cost extremely attractive.<\/p>\n\n\n\n

On the other hand, if it\u2019s related to the company\u2019s core competence, then investing in building up a team that can design and deploy models based on the company\u2019s proprietary data is the better way to go. It allows much more control and understanding of the process, though it does come with a high initial investment overhead.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

Just remember<\/h4><\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n
\n

All that matters to a business owner is: how will the output of your model be useful in the context of the business?<\/p>\n<\/blockquote>\n\n\n\n

Broadly speaking, machine learning is a next step up in automation. So in thinking this way, your model helps to increase revenue, or improves business efficiency by reducing costs.<\/p>\n\n\n\n

For instance, integrating a better recommendations system should lead to higher user satisfaction and engagement. Higher satisfaction leads to more subscriptions and organic promotion, which leads to higher revenue and lower churn.<\/p>\n\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n

As a data scientist, you have to help the business owner make the best decision on whether to build or buy, and that means assessing the specific pros and cons of either approach\u00a0in the context of the business<\/strong>.<\/p>\n\n\n\n

The points in this\u00a0article<\/a>\u00a0are very useful in approaching the \u201cbuild vs. buy\u201d question, and it also applies to machine learning production. There\u2019s a case for building it yourself, as argued\u00a0here<\/a>, if it is a core business competence.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

It all starts from the source<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n

Consider the search results for a user who searched for photos of puppies. Here we have an example from Unsplash, one from Shutterstock, and finally one from EyeEm, this time on a mobile device.<\/p>\n\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

We can see that while each user interface(UI) displays images in a wide selection. Suppose we want to use clicks on the image to train a model that gives better search results.<\/p>\n\n\n\n

Which image would you be drawn to first if you were a user?<\/p>\n\n\n\n

In the Shutterstock example, you may first be drawn to puppy on the right most side of the window, but in the EyeEm example, it could be the image in the center.<\/p>\n\n\n\n

It is clear that there are differences between each UI to cause differences in the distribution of the data collected.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n

Information retrieval literature has demonstrated that a\u00a0positional bias\u00b2<\/em>\u00a0and\u00a0presentation bias<\/em>\u00b3 exists when displaying results. It is entirely possible for users to click on the first ranked result, even if it\u2019s not relevant to their search query, simply because it\u2019s the first thing they see, and they might be curious about the result!<\/p>\n\n\n\n

We can see that the click data used to train search ranking models will have these biases in them, driven by how the UI was designed. So,<\/p>\n\n\n\n

\n

Know where and how the model will ultimately be used in the product<\/p>\n<\/blockquote>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

A data scientist and machine learning engineer\u00a0cannot build models in isolation from where the model will ultimately be used<\/strong>. A consequence of this is that a data scientist has to work with all manners of specialties who are involved in the product: the product manager, the designer, the engineers (frontend and backend) and quality assurance engineers.<\/p>\n\n\n\n

The cross-functional team work provides benefits, including briefing all team members about the limitations of the model, how that affects UI design and vice versa. Another is to find simple, reliable ways to solve problems, for instance, not solving an inherent UI (and user experience) issue using machine learning. Designers can also come up with ways to make the presentation of results from the model look amazing, as well as constraining user actions in the UI to limit the space of inputs that goes into a model.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n

A data scientist must build up domain knowledge about the product, including why certain UI decisions were made, and how they can impact your data downstream if product changes are made. The latter point is especially important, since it can mean your model is outdated very quickly when product changes are rolled out.<\/p>\n\n\n\n

Moreover, how the results from the model are presented is important so as to minimize bias in future training data, as we shall see next.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

Data collection is still challenging<\/h2>\n<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n

Analytics collection typically depend on engineers setting up analytics events in the services they are responsible for on various parts of the product. Sometimes, analytics are not collected, or collected with noise and bias in them. Analytics events can be dropped, resulting in missing data. Clicks on a button are not debounced, resulting in multiple duplicate events.<\/p>\n\n\n\n

Machine learning models are, unfortunately, typically downstream consumers of these analytics data, which means problems with measurements and data collection has a huge impact on the models.<\/p>\n\n\n\n

If there\u2019s one thing to remember about data it\u2019s that<\/p>\n\n\n\n

\n

Data is merely a byproduct of measurement<\/p>\n<\/blockquote>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

This means that it is important to\u00a0understand where the data is coming from, and how it is being collected<\/strong>. It is also then paramount that measurements are made correctly, with tests on the measurement system conducted periodically to ensure good data quality. It would also be prudent to work with an engineer and a data analyst to understand the issues first-hand because there\u2019s usually a \u201cgotcha\u201d in collected data.<\/p>\n\n\n\n

Stored data needs to be\u00a0encoded to suit the business domain<\/strong>. Let\u2019s take image storage for instance. In most domains, a lossy JPEG image may be fine to encode your stored images, but not in some domains where very high resolution is needed, such as using machine learning for detecting anomalies in MRI scans, for instance. It\u2019s important that these issues be communicated up-front to engineers who are helping you collect the data.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Machine learning in research differs from machine learning in the product in that\u00a0generally your biggest wins come from better data, not so much better algorithms<\/strong>. There are exceptions, though, such as deep learning improving performance metrics in many areas. Even then, these algorithms are still sensitive to the training data, and you need a whole lot more data.<\/p>\n\n\n\n

It is always worth convincing your stakeholder to continually invest in improving the data collection infrastructure<\/strong>.\u00a0<\/em>Sometimes, just even a new source of data to construct new features can boost model performance metrics. Not only does it help with better models, it helps with better tracking of the company\u2019s key metrics and product features overall. The data analysts you work with will be very thankful for that.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n

As an ex-Dropbox data scientist told me recently<\/p>\n\n\n\n

\n

Your biggest improvements usually come from sitting down with an engineer to improve analytics<\/p>\n<\/blockquote>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

Check your assumptions on the data<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n

Here is one story I experienced first-hand from the trenches.<\/p>\n\n\n\n

During the early days of incorporating learning-to-rank models to improve image search at\u00a0Canva<\/a>, we planned to use a set of relevance judgments sourced from Mechanical Turk. Canva has a graphic design editor tool that allows a user to search over 50 million images to include into their design.<\/p>\n\n\n\n

Basically, what this meant was collected sets of pairwise judgements: for each search term, raters are given a pair of images, one of them which they give a thumbs up to. An example pairwise judgement rating for the term\u00a0firefly\u00a0<\/em>is shown below. Although both are relevant, the image on the left is more preferred by the rater than the image on the right.<\/p>\n\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

We then trained a model using these pairwise judgements, and another model that solely relied on clicks from search logs. The latter has more data, but is of lower quality due to positional bias and errant clicks. Both models look great on offline ranking metrics, and a visual check of the results (i.e.,\u00a0<\/em>manually, using our eyes).<\/p>\n\n\n\n

In online experiments, however, the model that relied on these sourced relevance judgements had a very poor performance, tanking business metrics, compared to the control and the other model.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

What happened?<\/h3><\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n

As it turned out, we sourced the judgements\u00a0without<\/em>\u00a0the context of users searching for images to fit their design, not just on pure search relevance alone. This missing context contributed to a dataset that had a different distribution from the data distribution in the product.<\/p>\n\n\n\n

Needless to say, we were wrong in our assumption.<\/p>\n\n\n\n

Data distribution skew<\/em>\u00a0is a very real problem: even the team that implemented the Quick Access functionality in Google Drive at Google faced it\u2074. In this case, the distribution of data collected for their development environment did not match the data distribution of the final deployed environment, as the training data was not collected from a production service.<\/p>\n\n\n\n

\n

The data used to train your models should match the final environment where your model will be deployed<\/p>\n<\/blockquote>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n

A sample of other assumptions to check for are<\/p>\n\n\n\n

    \n
  1. the appearance of a\u00a0concept drift<\/strong>, where a significant shift in the data distribution happens (a very recent example is the COVID-19 outbreak messing with prediction models<\/a>),<\/li>\n
  2. the\u00a0predictive power of features<\/strong> going into the model as some features decay in predictive power over time, and<\/li>\n
  3. product deprecations<\/strong>, resulting in features disappearing.<\/li>\n<\/ol>\n\n\n\n

    These are partially solvable by having monitoring systems in place to check data quality and model performance metrics, as well as flag anomalies. Google has provided an excellent checklist on scoring your current production machine learning system\u00a0here<\/a>, including data issues to watch out for.<\/p>\n\n\n

    Remember, if a team at Google, whose maturity in deploying machine learning systems is one of the best in the world, had data assumption problems, you should certainly double-check your assumptions about your training data.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t

    Wrapping it all up<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\n

    Machine learning systems are very powerful systems. They provide companies with new options in creating new product features, improving existing product features, and improving automation.<\/p>\n\n\n\n

    However, these systems are still fragile, as there are more considerations about how it\u2019s designed, and how data is collected and used to train models.<\/p>\n\n\n\n

    I still have more to say on the topic, but these will do for now.<\/p>\n\n\n

    \n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    [1] Substitute with your favorite cloud provider.<\/p>\n\n\n\n

    [2] Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay,\u00a0Accurately Interpreting Clickthrough Data As Implicit Feedback<\/a>, In Proc. of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pp.154\u2013161 (2005).<\/p>\n\n\n\n

    [3] Yisong Yue, Rajan Patel, and Hein Roehrig,\u00a0Beyond Position Bias: Examining Result Attractiveness As a Source of Presentation Bias in Clickthrough Data<\/a>, In Proc. of the 19th International Conference on World Wide Web (WWW), pp. 1011\u20131018 (2010).<\/p>\n\n\n

    [4] Sandeep Tata, Alexandrin Popescul, Marc Najork, Mike Colagrosso, Julian Gibbons, Alan Green, Alexandre Mah, Michael Smith, Divanshu Garg, Cayden Meyer, Reuben Kan,\u00a0Quick Access: Building a Smart Experience for Google Drive<\/a>, KDD \u201917: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp.1643\u20131651 (2017)<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"

    Machine learning systems are very powerful systems. They provide companies with new options in creating new product features, improving existing product features, and improving automation. However, these systems are still fragile, as there are more considerations about how it\u2019s designed, and how data is collected and used to train models.<\/p>\n","protected":false},"author":875,"featured_media":9210,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[183],"tags":[92,850,518],"ppma_author":[3814],"class_list":["post-9209","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-ml","tag-machine-learning","tag-machine-learning-models","tag-train-models"],"authors":[{"term_id":3814,"user_id":875,"is_guest":0,"slug":"paul-tune","display_name":"Paul Tune","avatar_url":"https:\/\/www.experfy.com\/blog\/wp-content\/uploads\/2020\/08\/Paul-Tune-150x150.jpg","user_url":"http:\/\/www.canva.com","last_name":"Tune","first_name":"Paul","job_title":"","description":"Paul Tune is Machine Learning Engineer at Canva"}],"_links":{"self":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/9209","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/users\/875"}],"replies":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/comments?post=9209"}],"version-history":[{"count":6,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/9209\/revisions"}],"predecessor-version":[{"id":34288,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/9209\/revisions\/34288"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media\/9210"}],"wp:attachment":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media?parent=9209"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/categories?post=9209"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/tags?post=9209"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=9209"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}