service-streamer

ShannonAI/service-streamer is ranked 10,996 among these 8 peers; ENTERPILOT/GoModel is ranked 11,007.

Service Streamer Boosting your Web Services of Deep Learning Applications. • Highlights • Installation • Develop BERT Service in 5 Minutes • API • Benchmark • FAQ • • Made by ShannonAI • :globe_with_meridians: What is Service Streamer ? A mini-batch collects data samples and is usually used in deep learning models. In this way, models can utilize the parallel computing capability of GPUs. However, requests from users for web services are usually discrete. If using conventional loop server or threaded server, GPUs will be idle dealing

with one request at a time. And the latency time will be linearly increasing when there are concurrent user requests. ServiceStreamer is a middleware for web service of machine learning applications. Queue requests from users are sampled into mini-batches. ServiceStreamer can significantly enhance the overall performance of the system by improving GPU utilization. Highlights :hatching_chick: **Easy to use**: Minor changes can speed up the model ten times. :zap: **Fast processing speed**: Low latency for online inference of machine

learning models. :octopus: **Good expandability**: Easy to be applied to multi-GPU scenarios for handling enormous requests. :crossed_swords: **Applicability**: Used with any web frameworks and/or deep learning frameworks. Installation Install ServiceStream by using ，requires **Python >= 3. 5** : Develop BERT Service in 5 Minutes We provide a step-by-step tutorial for you to bring BERT online in 5 minutes. The service processes 1400 sentences per second. Text Infilling is a task in natural language processing: given a sentence

with several words randomly removed, the model predicts those words removed through the given context. BERT has attracted a lot of attention in these two years and it achieves State-Of-The-Art results across many nlp tasks. BERT utilizes "Masked Language Model (MLM)" as one of the pre-training objectives. MLM models randomly mask some of the tokens from the input, and the objective is to predict the original vocabulary id of the masked word based on its context

. MLM has similarities with text infilling. It is natural to introduce BERT to text infilling task. First, we define a model for text filling task bert_model. The function accepts a batch of sentences and returns predicted position results of the token. Second, utilize Flask to pack predicting interfaces to Web service. Third, encapsulate model functions through . Three lines of code make the prediction speed of BERT service reach 200+ sentences per second (16x faster). Finally, encapsulate models through Streamer

and start service workers on multiple GPUs. Streamer further accelerates inference speed and achieves 1000+ sentences per second (80x faster). API Quick Start In general, the inference speed will be faster by utilizing parallel computing. **ServiceStreamer** is a middleware for web service of machine learning applications. Queue requests from users are scheduled into mini-batches and forward into GPU workers. ServiceStreamer sacrifices a certain delay (default maximum is 0. 1s) and enhance the overall performance by improving the ratio of

GPU utilization. Start web server on multi-threading (or coordination). Your server can usually achieve 10x ( ) times faster by adding