Vadzim Tsitko

How to Find a Data Science Job in 2024 (with experience)

2024-04-13T00:00:00+00:00

At the end of 2023, I decided to change my job. I had worked previously in multiple companies as a Data Scientist (Machine Learning Engineer): an outsourcing firm, a small startup, and a medium-sized mobile applications publisher. But I had never worked before in big tech or similar companies, so my main focus was to find a job in one of those.

In this article, I will tell you about how I found a new job, how I prepared for the interviews, and what my success rate was. And because experience matters a lot, I will provide my background information below.

I am a Data Scientist with 4 years of experience.
I have a bachelor’s degree in Computer Science.
I was looking for a job in Europe because I am living there.

Preparation

In this part, I will tell you how I prepared for the interviews. Some parts could be relevant to any Software Engineering position, while other apply only to Data Scientists or Machine Learning Engineers. You can skip the parts you are not interested in.

Coding

Because my main focus was big tech companies, it made sense to improve my LeetCode skills. In university, I took multiple algorithms courses, read a couple of books on algorithms and completed one or two online courses. So, I was not a novice at all. But still, knowing general algorithms and solving LeetCode tasks in 20 minutes are quite different tasks.

I started solving LeetCode problems. My plan was to solve at least one per day. And I did so roughly for 3 months, with a one-week break for vacation. In the beginning, I spent around 3-4 hours a day. In the last month, I just solved the daily challenge as fast as I could.

As a result, I solved 300 tasks. I think it is much more than you actually need. But I loved the process.

You need theoretical knowledge to solve those questions. Again, I could be biased because I learned this stuff in university and by myself for a couple of years beforehand. But from the experience of my friends, solving the Udemy course Master the Coding Interview: Data Structures + Algorithms and reading Grokking Algorithms: An Illustrated Guide for Programmers and Other Curious People will be enough to begin. I did it too, just a little earlier.

But theory will not take you far. You need lots of practice. And for that, we all use LeetCode. From my standpoint, it will be enough to solve the whole Grind 75 or neetcode.io, which will take you a couple of weeks with a tight schedule.

I think the best approach will be to spend 30-40 minutes on each task. I always begin by drawing needed structures and conditions. This leads me to the initial solution, which then I optimize if needed. After that, I write the solution in pseudo-code on paper. And then transform it into my Python solution. This may sound like overkill. But in reality, the first step takes most of the time. If you have a drawing of your solution, then writing the code will be a piece of cake.

If you are not able to solve the task by yourself, watch the solution on the NeetCode YouTube channel, and then code the solution again by yourself. This way, you will memorize and understand the solution much better than simply copying it from the Solutions tab.

If some topic is not understandable to you, I recommend spending a couple of hours on YouTube. For example, the legendary channel of Abdul Bari. And I suggest implementing the new structure or algorithm in your programming language from scratch. It will solidify your base and understanding of the basic Computer Science topics.

And remember, you don’t need to be a genius to solve those tasks. You need patience, time, and consistency. I still can’t solve some of the medium tasks, and I skip hard tasks altogether. Don’t be hard on yourself, just stick to it, and you will succeed.

Data Science and Machine Learning

Those topics are really broad and everybody expects different things from those specialists. Some companies need to build matching systems, some LLM-powered tools, and some analytical dashboards. Therefore, it’s crucial to adapt according to the job description. But, there is general knowledge which you should know for most interviews.

I think Chip Huyen did an amazing job writing the Machine Learning Interviews Book. To prepare for the interview, I recommend reading this book and solving tasks in which you are not proficient. For example, I forgot probability theory. Working through tasks from this book, along with a few additional ones from the internet, allowed me to refresh my knowledge.

Also, going through the list of popular questions is helpful too, for example, Data-Science-Interview-Questions-Answers.

Machine Learning System Design

This topic is growing rapidly. But I still have some resources that I love.

First of all, these two books:

Designing Machine Learning Systems: An Iterative Process for Production-Ready Applications by Chip Huyen (again)
Machine Learning System Design by Valerii Babushkin and Arseny Kravchenko

These books are great; they are not about interviews but about the general approach to building ML Systems.

I also loved this article by Stefan Hosein. It provides you with a practical approach to handling this type of interview. Last, but not least, I do recommend ML System Design Interview videos from karpov.courses. These are originally in Russian, but in my opinion, they are the best videos on the topic. So I strongly suggest you watch them with English subtitles. And by the way, they are hosted by Valerii Babushkin, one of the authors of the Machine Learning System Design book.

Generally, reading articles and being curious about the field pays off in this interview.

Mock Interviews

These are crucial. You can test your interviewing abilities in a controlled environment. There are services for which you can pay, but I recommend using the help of your friends or reaching out to people on LinkedIn and asking them for help. You will be surprised how many people are willing to help you for free. They gain experience from it too!

Resume

There is an infinite amount of information on this topic. My general recommendations are:

Keep it to just one page.
Do not add your photo. I know you are pretty, but a picture takes up space.
Use a simple and minimalistic design.
Sort blocks by importance, from top to bottom.
Add not only technical but also business information. For example, “By building X, I improved retention by Y.”

Interviews

The interviewing process sucks. It is not the best proxy for your job performance, but it is the main one we use. I think there is a lot of information on how to prepare for interviews mentally.

My main advice for anybody is to gain experience. Do mock interviews, and interview for companies you are not interested in first. Don’t worry about rejections. Interviewing is a skill in itself. To become better, you need to practice.

With the companies you want to work for, you can significantly boost your chances using simple things.

First of all, try to have at least a couple of real interviews with other companies before you interview with the company of your dreams.

Second, research the company and the role. Try to think of tasks you can potentially solve in this company. Prepare answers to the basic behavioral questions and prepare your questions for the interviewer. Read the interview section on the company’s Glassdoor page.

Third, apply to the companies using a referral system. For example, via an acquaintance of yours. Or again, find a person who works there on LinkedIn and ask them to refer you. Most people will refer you without any problems. They want to get a bonus if you wind up in their company.

Fourth, don’t freak out about rejections. You can reapply after half a year in most cases. And then you will be much more experienced and prepared.

Duration

Searching for a job takes time. I think you should expect the process to take 3 to 6 months, from the application to the first day in the company. Typically, with smaller companies or startups, it takes less time, while with big tech companies, the process is longer.

Questions

Do not be afraid to ask questions. If anything, it will make things better, not worse.

Before each interview, ask your HR questions about the interview. What will this interview be about? How should I prepare? Can I look at the LinkedIn profile of the interviewer? These questions will allow you to prepare much better. You will know what to expect, which will take some of the anxiety off you.

Ask questions during the interview if you think that you didn’t understand your task fully. Sometimes interviewers ask ambiguous questions and expect you to ask clarifying questions in response. Or they simply weren’t able to formulate the question correctly.

And finally, at the end of each interview, you will be able to ask your questions. Use this time wisely. Ask questions that are worrying you, about the role and the company. Before the interview, prepare a list of general questions you will ask every company and a separate list for the company you are interviewing with. This way, you will learn something useful about the company. And, which is also important, the interviewer will notice that you are prepared and that you are interested in the company. If you can’t come up with your list, use the reverse-interview GitHub repository as your muse.

Overall, questions are extremely important. And sometimes asking the right questions will make a lot of impact.

My Latest Experience

Below, you can see the table I used to track my interviewing progress with all the companies in 2024. Looks scary, doesn’t it? Most of those are rejections or ignoring of my applications. If you like the table, you can duplicate it from here.

Let’s look at the pie plot below. Am I a Data Scientist or not?

Again, it looks scary. My hit rate is 3.6%, just 3 out of 83 companies. It would be fair to say that with companies under the Stopped label, I was pretty deep in the process. However, by this stage, I had already accepted an offer. So, I didn’t want to waste people’s time on myself.

In the table below, you can find the absolute numbers if you are interested.

Offer	3
Stopped	6
No answer	32
Rejection	42
Total	83

Luckily, I got an offer from the company I wanted to work for — Revolut. I visited their meet-ups and used their application daily. A colleague from one of my previous jobs works there, so he referred me for the position I wanted. And I was able to pass all the interviews and got the role! But without all the preparation, I wouldn’t have been able to do this.

Conclusion

For the most part, finding a job is a numbers game. But, using an analytical approach, you will maximize your chances of getting offers from the companies you want to work for.

Take your time. Learn from your failures. Be prepared and practice a lot. Hard work will pay off sooner or later.

And a huge thanks to Maria Zakharevich and Anton Masiukevich for their invaluable assistance with this article.

What I’ve Learned After 200 Random Coffees

2023-12-04T00:00:00+00:00

Many of us have been on first dates. For the first 10 minutes, it’s usually awkward, but then you find common topics to discuss, share a few jokes, and enjoy a coffee. As a result, you have a great time and learn something new. Even if you don’t plan to see that person again, it’s still a valuable experience.

Random Coffees are quite similar. You meet with a stranger, perhaps through your company’s random coffee program, an online community, or a mobile app. You have an hour to talk about anything you want: careers, sports, books, films, dogs, or even relationships.

And the best thing is, it’s periodic—once a week, for example. This lets you meet a ton of new people and establish a wonderful habit.

Why I Started Doing Random Coffees

I began participating in Random Coffees four years ago when I was a Junior Machine Learning Engineer at a small company. I realized that to advance professionally, I needed to hone my listening and communication skills. And as they say, the best way to learn is by doing. Random Coffees became my method of choice. I signed up for the activity within a professional community.

I can still recall my first Random Coffee—it was with an ML Engineer from Snapchat. Over the course of an hour, we discussed the latest ML technologies, the inner workings of big tech companies, interview strategies, and a wide array of other topics. He even suggested I apply for their trainee program—something I never got around to doing, of course.

After the call, I felt completely energized and motivated to improve as a professional. From that moment, I began to engage in Random Coffees with more or less consistency—certainly with greater regularity than my dating life.

I’ve met numerous fascinating individuals from various firms, including Google, Microsoft, JetBrains, Yandex, and many more. Our conversations ranged over a multitude of topics, from technology to travel and art.

Once, I talked to a person who, years earlier, had encouraged me in a chat to share my code with the world. He didn’t remember it, but for me, it was a significant moment.

What I Learned

Naturally, I’ve picked up insights on some new technologies and ML papers, but that isn’t the heart of what I’ve gained. From these meetings, I’ve learned something far more important: how to engage in open communication with others, overcome shyness during first-time interactions, and keep conversations flowing even when topics seem scarce. These are skills you can’t lean from books – they only come with practice.

This experience has been valuable during job interviews and in my daily life. It’s now much easier for me to make a phone call addressing a problem with my documents or to initiate a chat with a fellow dog owner on a walk.

Certainly, the first calls were challenging, especially when the person on the line was shy or having a bad day. I’ve had calls where the other person was so unresponsive that getting any understandable answer felt impossible. Yet, it’s exactly these situations that provide the richest lessons.

After one of those extremely hard calls, where the person couldn’t share even two words, I ended up exhausted and soaked in sweat. The lesson I learned from that conversation: sometimes it’s best to end things early.

How to Maximize the Benefits

But my greatest advise is: just show up. This way sooner or later you will became better at this ancient skill of communicating.

To begin with, simply show up. Don’t constantly reschedule or ignore the other person. Better yet, take the initiative and reach out first. And do it consistently. Aiming for a weekly meetings is ideal.

Once you’re more accustomed to the process, spend 15 minutes researching the other person’s LinkedIn or Instagram profiles. This preparation will help you find potential conversation starters. For example, discovering that the individual enjoys skydiving can lead to a great topic of conversation.

Additionally, prepare some general questions to ask everyone you meet. My go-to question is, “What’s the best book you’ve read ever?” As an constant reader, I often find myself in a twenty-minute discussion about literature alone.

Make notes about each person you meet. Maybe it is a little creepy. But this way, you can use this connection in the future. For instance, someone working at Microsoft might be willing to refer you to their HR department, even if you’ve only met once.

As you grow more confident, consider reaching out directly to individuals on LinkedIn who interest you, and propose a call. This proactive approach can provide you with great insights into a company you’re eager to join or a technology that fascinates you.

By using this approach, I connected with someone whose YouTube courses taught me Deep Learning years ago. Our dialogue was somewhat life-changing.

But the most important piece of advice I can offer you is this: just show up. It will inevitably boost your skills in the art of communication.

“Should I Do It?”

Yes. Without a doubt. 100%.

This skill will benefit every area of your life, and it only requires an hour a week. Many large companies and communities offer this activity. So, why not join?

How to use Diffusion Models for Data Augmentations

2023-08-28T00:00:00+00:00

Data augmentations are an essential part of training any Deep Learning algorithm. There are numerous options to choose from, such as flips, color jittering, MixUps, crops, and more. However, in the era of modern Machine Learning, we can explore the use of Generative AI for augmentations.

In this article, we will look into relevant research papers and benchmarks, compare various approaches, and determine whether diffusion models are effective tools for data augmentation.

Approaches to Data Augmentation with Diffusion Models

Data augmentation is a technique that helps reducing overfitting by enhancing our dataset with modified versions of existing samples. For instance, when training an image classification model, we can apply random horizontal flips to each sample during the training process. Let’s explore which techniques can be employed to enable data augmentation with diffusion models.

Boomerang

In the paper titled “Boomerang: Local sampling on image manifolds using diffusion models”, researchers use Stable Diffusion to make small changes to each dataset sample. They apply img2img with a limited number of diffusion steps, resulting in images that appeared similar but were actually different. Let’s take a look at an example.

The paper reports increased accuracy on CIFAR-10 and ImageNet-200 datasets. However, it’s not entirely clear from the paper what augmentations were used for the baseline, thus making the results less representative. In this article, we will explore another paper that compares Boomerang with other approaches.

Prompt Conditioning

This method involves generating images of the required class using Stable Diffusion. We can condition the model using prompts that include the name of the class. For instance, prompts like “a photo of the n” or “a HQ image of the n” can be used. Additionally, instead of using the vanilla Stable Diffusion 1.5, we can use a fine-tuned version, which may yield more photorealistic results. For the images presented below, I used fine-tune Deliberate.

prompt: a photo of the leopard

prompt: a HQ image of the leopard

Textual Inversion

Textual Inversion is a technique that allows us to teach a diffusion model novel concepts using only a small number of example images. The concept was introduced in the paper “An Image is Worth One Word”. With this approach, we teach SD models on concepts from our own dataset. This becomes particularly valuable when our domain is not properly represented in the datasets on which SD was initially trained. For example, if we are training a model to classify various types of furniture or exotic flowers, Textual Inversion can help us in achieving better results.

The downside is that we need to train textual inversion for every concept in our dataset. For large datasets like ImageNet, which can contain thousands of concepts, this approach can become time and resource-consuming.

Also, we can fine-tune the entire model on our images, or we can fine-tune just the text encoder, which, in the case of SD, is CLIP. However, these techniques suffer from the same problems.

Comparison

For the comparison, we will use data from the amazing paper “A data augmentation perspective on diffusion models and retrieval.” I highly encourage you to read this paper, as it provides a nice overview of the field.

Baseline

To simulate the training pipeline when training data is scarce, the authors sampled only 10% of the ImageNet-1000 dataset.

Upper-bound Model

To establish an upper-bound model, they used the same training routine but with 20% of the ImageNet-1000 dataset. This allows us to compare not only how diffusion augmentations perform compared to the baseline, but also to compare the results of diffusion augmentations to the model that has more real-world data.

Nearest Neighbor Retrieval

We can retrieve samples from the Stable Diffusion training dataset, Laion 5B. We can do this based on their CLIP similarity to the classes from ImageNet-1000. This way, we will be able to see if it is reasonable to use Stable Diffusion for augmentations, or if it is better to retrieve data from the Stable Diffusion training dataset. For your projects, you can use tools that simplify the process of CLIP retrieval.

The authors evaluated the entire ImageNet validation split. They applied random resizing and cropping augmentations to each method mentioned above. The baseline model utilized a ResNet-50 classifier trained on this data. For each method, they used the original 10% of the ImageNet as the training data and supplemented it with additional augmented data. The diffusion augmentations were performed using Stable Diffusion 1.4.

Let’s dig into the results.

Augmentation method	Top-1 accuracy (%)
10% ImageNet	57.2 ± 0.2
20% ImageNet	70.2 ± 0.3
Boomerang	56.3 ± 0.3
Prompt Conditioning (CLIP prompts)	60.9 ± 0.2
Textual Inversion	61.0 ± 0.4
Nearest Neighbor Retrieval	62.6 ± 0.1

As expected, the model trained with 20% of ImageNet performed the best. It proves that it is always better to collect more real-world data. Interestingly, Bommerang performed poorly, even worse than the model trained on 10% of ImageNet. My guess is that this is because Bommerang doesn’t significantly change the data, which could lead to overfitting. Below, you can see how little information changed in the images.

Maybe if authors applied more diffusion steps and combined this with Prompt Conditioning, it would yield better results.

The most interesting thing from the paper is that Nearest Neighbor Retrieval gives the best result among all discussed techniques except for 20% ImageNet-1000. And, once again, it proves that real-world data rocks. And that the CLIP retrieval is a quite powerful technique.

An important thing to mention in this article is that the authors used Stable Diffusion 1.4 for their experiments. However, at the time of this writing, SD 1.5 and 2.1 are already available, and the release of SDXL 1.0 is just around the corner. The generation quality of these newer models is significantly better than SD 1.4. Considering this progress, it is likely that in the upcoming months, we will see new research papers that more successfully utilize diffusion models for augmentations or even zero-shot tasks.

Conclusion

Using diffusion models is a beneficial way to enrich your data, particularly in tricky cases where Textual Inversion or similar techniques can be applied. However, it is even more advantageous to obtain additional real-world data. To achieve this, you can leverage CLIP and utilize a large-scale dataset, such as Laion-5B.

Let’s Understand Stable Diffusion Inpainting

2023-08-16T00:00:00+00:00

In this article, we will explore the simplest approach to use Stable Diffusion for image inpainting. We all know, that the best way to grasp a concept is by implementing it. We will enhance the existing StableDiffusionImg2ImgPipeline from diffusers to solve inpainting tasks.

👨🏻‍🎨 It is better to use either StableDiffusionInpaintPipeline or StableDiffusionInpaintPipelineLegacy from the 🤗 Diffusers library for your real-world tasks. These options are more comprehensive. The first one is specifically designed to work with SD models that have been fine-tuned for inpainting tasks.

All the code, images and requirements can be found in my GitHub repository diffusers-inpainting.

How Stable Diffusion Works

Let’s begin with a brief theory lesson. Stable Diffusion is a diffusion model that generates images by operating on the latent representations of those images. The algorithm looks like this:

Stable Diffusion retrieves the latents of the given image from a variational autoencoder (VAE).
It uses CLIP to obtain embeddings of the given prompt.
The diffusion process takes place using a UNet-like model, utilizing the latents from the first step and the prompt embeddings from the second step.
Finally, the result of the diffusion process is decoded using the VAE.

If we want to focus only on text-to-image generation without the need for image-to-image, we can skip the first step and begin the diffusion process directly from noise. This scheme is oversimplified, but it accurately captures the main idea.

But what is the diffusion process? It is a process in which a UNet-like model receives latents and prompt embeddings as inputs. Then, it adds noise to the latents or starts directly from noise if it is not an image-to-image task. The amount of noise added is determined by a hyperparameter.

The UNet, given initial latents and prompt embeddings, predicts noise, which is then subtracted from the input. The resulting output is passed through the UNet again, making it an iterative algorithm. So at each diffusion step, model “cleans out” noise from the latents. After several dozen iterations, we obtain VAE latents that can be decoded into an image. The number of iterations is also a hyperparameter.

This process is depicted in the diagram below. If you want to understand this process better, I advise you to read High-Resolution Image Synthesis with Latent Diffusion Models paper. And go threw StableDiffusionImg2ImgPipeline implementation.

How To Make Inpainting Work

There are several approaches to incorporate Stable Diffusion (SD) for inpainting tasks. One such method is to fine-tune the original SD model. RunwayML has implemented this approach, making runwayml/stable-diffusion-inpainting checkpoint:

First 595k steps regular training, then 440k steps of inpainting training at resolution 512x512 on “laion-aesthetics v2 5+”. For inpainting, the UNet has 5 additional input channels (4 for the encoded masked-image and 1 for the mask itself) whose weights were zero-initialized after restoring the non-inpainting checkpoint.

But lots of finetunes were not trained on inpainting task at all. So how can we use them for this task?

At each iteration step, we can substitute the area that is not under the mask with the original image latents plus the amount of noise needed for this step. The area under the mask we will leave untouched. This way, we will make sure SD only changes the masked area. And at the same time, SD will know about the area that needs to stay the same.

Let’s see how we can update StableDiffusionImg2ImgPipeline, to work this way. At each step I will provide GitHub link to the needed line in diffusers-inpainting repository.

Add new parameter mask to the __call__ method

 mask_image: Union[torch.FloatTensor, PIL.Image.Image] = None,

Check, that mask_image is not None

 if mask_image is None:
 		raise ValueError("`mask_image` input cannot be undefined.")

Preprocess the mask
```
 mask = prepare_mask(mask=mask_image)
```

Resize input mask to the size of the latents, which VAE outputs

 height, width = mask.shape[-2:]
 mask = torch.nn.functional.interpolate(
     mask, size=(
 				height // self.vae_scale_factor, 
 				width // self.vae_scale_factor
 		)
 )

At the end of each diffusion step substitue area which is not under the mask, with original input latents plus noise needed for this step. Area under the mask leave untuched.

 # Adding noise to original latents
 init_latents_proper = self.scheduler.add_noise(
     init_latents, noise, t
 ).to(device)
    
 # Using mask
 mask = (mask > 0.5).to(prompt_embeds.dtype)
 latents_with_noise = (
     mask * latents_with_noise + (1 - mask) * init_latents_proper
 )

That’s it. With this setup, we will make SD generate something inside the masked area. The unmasked area will be left untouched because we substituted it with the original latents at each diffusion step. And because SD can change only the area under the mask, it will force itself to make it consistent with the whole image. At the same time, it will try to follow the prompt.

Examples

Below you can find the result of this algorithm, generated with the model redstonehero/Yiffymix_Diffusers, which doesn’t have any inpainting checkpoint.

Gif below shows denoising process for this image

This algorithm works and can give decent results. However, it performs poorly when we want to simply remove the object from the masked area. Let’s compare the performance of runwayml/stable-diffusion-v1-5 using our pipeline with runwayml/stable-diffusion-inpainting using the StableDiffusionInpaintPipeline. It’s important to note that runwayml/stable-diffusion-inpainting was specifically trained for the inpainting task.

To remove an object from the image, let’s provide an empty prompt to the model. With an empty prompt, the model will attempt to make the masked area as consistent with the rest of the image as possible.

So, we can see that our algorithm failed, but SD inpainting performed quite well. Additionally, a model specifically fine-tuned on the inpainting task will be able to produce better and more consistent results for text-guided image inpainting.

Conclusion

The algorithm we have implemented today can be a good choice if there is no inpainting model available for fine-tuning. However, if there is an inpainting model specifically trained for the task, it is undoubtedly better to use it.

If this article was useful, don’t forget to clap and leave a star in the diffusers-inpainting repository. If you have any questions, feel free to reach out to me on LinkedIn. I will gladly help! 😁

How to Deploy Neural Networks on iOS

2023-02-05T00:00:00+00:00

You can find the slides for this article here

Let’s assume we have a problem that requires a solution through an ML algorithm. To solve this problem, we generally need to follow these steps:

Analyze the problem
Collect, clean, and preprocess the data
Build the ML algorithm that solves the problem
Deploy the algorithm
Monitor the results

While we often find the 3rd step exciting, with all the reading and training, we tend to forget about the crucial step of deployment. Deployment is a critical component of any ML product, and we should plan for it from the early stages of the project.

In this article, I’ll provide a general overview of deploying Neural Networks models on iOS devices. But first, why do we need to deploy the model directly on the device instead of using a server?

Server-side vs Client-side Deployment

In general, there are two solutions for deploying Neural Networks models.

The first solution is to deploy the model on a server. This can be done using frameworks such as ONNX, OpenVINO, TensorRT, etc. This approach is flexible, as we can change the model without modifying the client. Additionally, we are not restricted by the computational resources of the device.

However, this solution requires an internet connection on the client side and incurs additional costs for server maintenance.

The second solution is to deploy the model directly on the device. This approach is cost-effective and provides complete privacy. There are no scalability issues with this approach, and our server costs won’t skyrocket if our application becomes popular.

The drawback of this approach is that mobile devices have limited computational resources, and some models may be too heavy to run on the device. Furthermore, some devices may not have adequate support for neural network operations.

In conclusion, the right approach depends on the problem. In my experience, it’s best to deploy models on the device whenever possible, and only use a server when it is not possible or a more flexible solution is required.

Deployment Frameworks

When training Neural Networks models, most developers use PyTorch or TensorFlow. However, deploying these models on iOS devices requires a different approach.**

PyTorch

PyTorch offers TorchScript for deployment on iOS devices. While there are examples of apps on the PyTorch website, this approach comes with several challenges:

TorchScript doesn’t support the Apple GPU or Apple Neural Engine.
There is no native support for TorchScript on iOS.
Developers need to write Objective-C or Objective-C++ wrappers for TorchScript.

If you want to explore this approach further, you can visit the PyTorch iOS deployment page for more information.

TensorFlow

TensorFlow allows for deployment on iOS devices using TensorFlow Lite. While TensorFlow provides examples of apps, it still faces similar challenges as PyTorch:

There is no native support for TensorFlow Lite on iOS.
Developers need to write Objective-C or Objective-C++ wrappers.
TensorFlow Lite has support for iOS GPU, but this support is limited and available only in a nightly version. The number of supported operations is also limited.

You can find more information about TensorFlow Lite deployment here and here.

CoreML

Core ML is an Apple framework that enables the integration of machine learning models into iOS apps. It is optimized for on-device performance, making use of the CPU, GPU, and Neural Engine, while keeping its memory footprint and power consumption low. To learn in-dep about Core ML, I recommend visiting this page.

When a Core ML model is added to a Xcode project, it automatically generates interfaces for the model. The framework supports image inputs and outputs and makes it easy to integrate pre- and post-processing.

So, using Core ML is the best option for integrating machine learning models into iOS apps. But during the development of these models, we typically use PyTorch or TensorFlow, not Core ML. What do we do then?

coremltools

To convert these models to the Core ML format, we use the coremltools framework. This framework can convert machine learning models from various third-party libraries, including PyTorch, TensorFlow 1 or 2, scikit-learn, and XGBoost.

For PyTorch models, the conversion process involves first converting the model to TorchScript, and then to Core ML. There are various input and output options that can be selected, such as setting the input to an image. This makes it possible for an iOS developer to pass the image to the model directly from the camera without having to perform any additional conversions.

Quantization is another option available during the conversion process. The model can be quantized to 16 bits, or even from 1 to 8 bits. However, the lower the number of bits, the greater the risk of reduced accuracy. The extent of the loss in accuracy depends on the specific model and may vary. In general, quantizing to 16 bits should be safe and is unlikely to have a significant impact on accuracy.

While the process may seem simple, there are always details that can make things more challenging. Core ML does not support all operations, particularly newly introduced ones, which can make it difficult to convert state-of-the-art models. However, with each new version, the situation is improving, although coremltools can still lag behind.

Custom layers

You can write your own custom operations, but the official documentation suggests using this option only as a last resort. Writing a custom operation requires implementation in Swift or Objective-C and separate implementations for CPU and GPU, using different frameworks.

It is often better to use a different model or retrain your existing model with layers that are already supported by Core ML. You can also try converting your model to Core ML before training, deploying it on iOS, and comparing the results with PyTorch. If everything checks out, then you can proceed with training. For a practical example, check out this blog post.

Processing units

Modern iOS devices can run neural networks on the CPU, GPU, and Neural Engine.

The CPU and GPU are well understood, but what is the Neural Engine?

The Apple Neural Engine (ANE) is a type of Neural Processing Unit (NPU) that accelerates neural network operations, such as convolutions and matrix multiplies. Unlike a GPU, which accelerates graphics, an NPU accelerates neural network computations. You can see the ANE and GPU cores as separate areas on the A12 Bionic chip.

The ANE isn’t the only NPU on the market, as other companies have developed their own AI accelerator chips as well. A well-known NPU besides the Neural Engine is Google’s TPU.

The first iteration of the Apple Neural Engine was introduced in the A11 chip, which was found in the iPhone X in 2017. It had a maximum processing power of 0.6 teraflops in half-precision and was utilized to efficiently run on-device ML features such as Face ID and Memoji.

In 2021, the fifth-generation 16-core ANE has advanced significantly, boasting 26 times more processing power, or 15.8 TFlops.

Core ML automatically selects the device (CPU, GPU, or Neural Engine) on which your model will run. However, it allows you to limit the device selection by specifying the following options:

CPU only
CPU and GPU
All (Neural Engine, CPU, and GPU)

For a comparison of performance for neural networks on CPU, GPU, and ANE, see this blog post. If you want to dive into the technical details of the Apple Neural Engine, visit this GitHub repository.

Conclusion

In conclusion, deploying a neural network on iOS can be a challenge, but with the right approach and tools, it can be done successfully. Start by testing the conversion to CoreML, as many state-of-the-art networks may not be convertible. Consider using an older algorithm that solves your problem and converts to CoreML or replacing non-convertible operations with convertible ones.

In most cases, a few percentage differences in the target metric may not be worth spending weeks trying to solve conversion problems. When you have found an architecture that supports CoreML, you can then train it for your specific task.

Quantizing your model to half-precision and changing input and output nodes to ImageType instead of FloatType can also improve inference speed and save time for iOS developers. If speed is a concern, ensure that your model uses the GPU or ANE. If it doesn’t, use XCode utilities for debugging.

Make Your Python Code Cleaner in One Easy Step

2022-10-24T00:00:00+00:00

Everybody knows that code needs to be understandable and easy to read. But at the same time, everybody forgets to write typings, sort imports, or follow PEP8. There is a solution — pre-commit.

This tool will trigger different hooks every time you commit your code. So, let’s integrate it into our project.

Installation

Let’s install pre-commit into the environment

pip install pre-commit

Then we need to add two files to the root directory of your project

.pre-commit-config.yaml

exclude: _pb2\.py$
repos:
- repo: https://github.com/psf/black
  rev: 22.3.0
  hooks:
    - id: black
      args: [ --skip-string-normalization ]
-   repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.0.1
    hooks:
    - id: check-docstring-first
    - id: check-json
    - id: check-merge-conflict
    - id: check-yaml
    - id: debug-statements
    - id: end-of-file-fixer
    - id: trailing-whitespace
    - id: requirements-txt-fixer
- repo: https://github.com/pre-commit/pygrep-hooks
  rev: v1.9.0
  hooks:
  - id: python-check-mock-methods
  - id: python-use-type-annotations
- repo: https://github.com/pre-commit/mirrors-mypy
  rev: 'v0.910'
  hooks:
  - id: mypy
    args: [--ignore-missing-imports, --warn-no-return, --warn-redundant-casts, --disallow-incomplete-defs]
    additional_dependencies: [types-all]
- repo: https://github.com/PyCQA/isort
  rev: 5.9.3
  hooks:
    - id: isort
      args: [ --profile, black, --filter-files ]

pyproject.toml

[tool.black]
line-length = 88
target-version = ["py38"]

[tool.isort]
profile = "black"
multi_line_output = 3

And now, you have to install the hooks that you’ve added.

pre-commit install

Usage

git commit -m "new brave code"

Yep, that’s it. Now, every time you’ll commit your code, those hooks will trigger. And automatically:

sort imports
format code for PEP8
check the correctness of your yaml and json files

And mypy will not pass your code if it’s not statically typed. So you’ll have to use typings in the right way.

Explanation

In the .pre-commit-config.yaml file, you specify which hooks you will use. For example, in the provided file, we use black code formatter of version 22.3.0. And ask not to change ‘ to " using a --skip-string-normalization flag.

In the pyproject.toml you specify parameters for those individual hooks. For example, you set the maximum line length to 88.