Understand Random Undersampling, Oversampling, SMOTE, ADASYN, and Tomek Links — Class imbalance occurs when one class in a classification problem significantly outweighs the other class. It’s common in many machine learning problems. Examples include fraud detection, anomaly detection, and medical diagnosis. The Curse of Class Imbalance A model trained on an imbalanced dataset perform poorly on the minority class. At best, this can cause loss…

Automate data science tasks with ChatGPT — I rated 60 ChatGPT functions for Data Science. Use these prompts and ask ChatGPT to write, and explain code, optimize data science code. It can also explain data science concepts, suggest ideas, and troubleshoot problems. How to Read this The table of content contains all 60 prompts. Click on the links in the table…

Data-centric AI without manually relabeling your data — I used an open-sourced library, cleanlab, to remove low-quality labels on an image dataset. The model trained on the dataset without low-quality data gained 4 percentage points of accuracy compared to the baseline model (trained on all data). Improving data quality sounds easy enough. It’s essentially identifying and rectifying wrong…

How to (and why you shouldn’t) use it — Here are two pain points of data scientists: Pain Point 1: Limited time in the data science lifecycle Data scientists have to prioritize. This may mean spending more time on understanding the business problem and identifying the most appropriate approach rather than focusing solely on developing machine learning algorithms. Pain point 2: Machine learning modeling can be time-consuming

Parameterize notebooks so you can programmatically run them — You’ve trained your machine learning model in a Jupyter notebook. Now, you want to run that model on data that comes in daily. Day in, day out, you create a new copy of the same notebook and run it. …

Stop waiting on pandas operations. Parallelize them. — One of the major limitations of Pandas is that it can be slow when working with large datasets, particularly when running complex operations. This can frustrate data scientists and analysts who need to process and analyze large datasets in their work. There are a few ways to address this issue…

Faster big-data analysis workflows with an open-source library — If you’re a data scientist working with large datasets, you must have run out of memory (OOM) when performing analytics or training machine learning models. That’s not surprising. The memory available on a desktop or laptop computer can easily exceed large…

Catch expensive mistakes early with nose2 and parameterized tests — You’ve deployed a new machine learning model at work. You can finally enjoy the weekend, you thought to yourself. Little did you know that an imminent storm of errors is about to tear down your model and ruin your weekend. Why does that happen? Insufficient error checking. Data scientists are…

Stop rewriting the same code for model selection and hyper-parameter search — Let’s face it — model training is extremely time-consuming. What if you could automate it? Meet TPOT, your data science assistant. It saves you time and effort in looking for the most optimized machine learning pipeline. Give TPOT the data, and it will give you the code for the most…