Complete-Life-Cycle-of-a-Data-Science-Project
CREDITS:All corresponding resources
MOTIVATION:Motivation to create this repository to help upcoming aspirants and help to others in the data science field
https://www.theinsaneapp.com/2021/03/how-to-build-machine-learning-project.html
**** If you like my work. please buy me a coffee it motivate me -> https://www.buymeacoffee.com/achuthasubhash?new=1 ****
Business understanding
1.Data collection
Data consists of 3 kinds
a.Structure data (tabular data,etc...)
b.Unstructured data (images,text,audio,etc...)
c.semi structured data (XML,JSON,etc...)
variable
a.qualitative (nominal,ordinal,binary)
b.quantitative(discrete,continuous)
https://www.chi2innovations.com/blog/discover-data-blog-series/data-types-101/
database scraping data from websites purchasing data data from surveys data, sensors, cameras, apis etc.
cleanlab https://l7.curtisnorthcutt.com/cleanlab-python-package https://github.com/cgnorthcutt/cleanlab https://github.com/cgnorthcutt/label-errors https://github.com/cgnorthcutt/rankpruning https://github.com/subeeshvasu/Awesome-Learning-with-Label-Noise
Measure Data Quality ydata-quality https://github.com/ydataai/ydata-synthetic https://towardsdatascience.com/how-can-i-measure-data-quality-9d31acfeb969
a.Web scraping best article to refer-https://towardsdatascience.com/choose-the-best-python-web-scraping-library-for-your-application-91a68bc81c4f
https://www.kdnuggets.com/2021/02/6-web-scraping-tools.html
https://www.bigdatanews.datasciencecentral.com/profiles/blogs/top-30-free-web-scraping-software
https://towardsdatascience.com/6-web-scraping-tools-that-make-collecting-data-a-breeze-457c44e4411d
https://medium.com/analytics-vidhya/master-web-scraping-completly-from-zero-to-hero-38051423256b
1.Beautifulsoup https://www.freecodecamp.org/news/how-to-scrape-websites-with-python-and-beautifulsoup-5946935d93fe/
mechanicalsoup https://analyticsindiamag.com/mechanicalsoup-web-scraping-custom-dataset-tutorial/
2.Scrapy,PyScrappy,Pandas Datareader,Instaloader,lxml
3.Selenium https://www.freecodecamp.org/news/better-web-scraping-in-python-with-selenium-beautiful-soup-and-pandas-d6390592e251/
4.Request to access data
5.AUTOSCRAPER - https://github.com/alirezamika/autoscraper https://www.youtube.com/watch?v=9BQ353Yu1D0 https://www.analyticsvidhya.com/blog/2021/04/automate-web-scraping-using-python-autoscraper-library/
scrapeasy Scrape Any Website in Seconds with One Line of Code https://github.com/joelbarmettlerUZH/Scrapeasy
Scrap Images From E-Commerce Website Using AutoScraper https://www.analyticsvidhya.com/blog/2021/05/scrap-images-from-e-commerce-website-using-autoscraper-library/
amazon auto scraper library https://webautomation.io/
Listly https://www.listly.io/r/stdfr
FiftyOne Now easier to download and evaluate https://towardsdatascience.com/googles-open-images-now-easier-to-download-and-evaluate-with-fiftyone-615ce0482c02
webbot https://pypi.org/project/webbot/
gazpacho https://github.com/maxhumber/gazpacho
html_scraper_streamlit_app https://www.youtube.com/watch?v=6U5xJ3mXRKA&feature=youtu.be
6.Twitter scraping tool (𝚝𝚠𝚒𝚗𝚝 or tweepy or tweetlib)-https://github.com/twintproject/twint
twitterscraper https://www.youtube.com/watch?v=MpIi4HtCiVk
twython https://github.com/ryanmcgrath/twython
twarc https://github.com/DocNow/twarc https://scholarslab.github.io/learn-twarc/01-quick-start.html
snscrape extract twitterr data https://github.com/JustAnotherArchivist/snscrape
Scweet A simple and unlimited twitter scraper https://github.com/Altimis/Scweet
GetOldTweets3,GoogleNews,snscrape,GetOldTweets3
Scrape Twitter for Tweets https://github.com/taspinar/twitterscraper
HAR File Web Scraper https://stevesie.com/har-file-web-scraper https://www.youtube.com/watch?v=LcqVDfueb8g
https://analyticsindiamag.com/complete-tutorial-on-twint-twitter-scraping-without-twitters-api/
https://developer.twitter.com/en/docs
pytrends https://medium.com/nerd-for-tech/scraping-data-from-online-platforms-to-enhance-time-series-forecasts-6eec3c68636d
Scraping Instagram -instaloader https://thecleverprogrammer.com/2020/07/30/scraping-instagram-with-python/
Instascrape
Scrape LinkedIn Profiles with ProxyCurl API
Reddit Dataset Using PSAW and PRAW in Python
Scraping Reddit using Python Reddit API Wrapper (PRAW)
Scrape Wikipedia wikipedia https://www.thepythoncode.com/article/access-wikipedia-python
patang - Scrape Product details from eCommerce Sites with Puppeteer and DOM String https://www.youtube.com/watch?v=3sgxRmyOuXs
Download Wikipedia https://www.wikidata.org/wiki/Wikidata:Main_Page https://www.youtube.com/watch?v=hC1rY4lRY0s https://towardsdatascience.com/an-efficient-way-to-read-data-from-the-web-directly-into-python-a526a0b4f4cb
Web Scraping to Create a CSV File https://thecleverprogrammer.com/2020/08/08/web-scraping-to-create-csv/
Amazon Web Scraper, Amazon Auto Scraper
7.urllib
8.pattern
9.Octoparse Easy Web Scraping https://www.octoparse.com/
prowebscraper https://prowebscraper.com/features
Web scraper https://chrome.google.com/webstore/detail/web-scraper-free-web-scra/jnhgnonknehpejjnehehllkliplmbmhn?hl=en
ParseHub https://www.parsehub.com/ https://analyticsindiamag.com/parsehub-no-code-gui-based-web-scraping-tool/
PyScrappy https://github.com/mldsveda/PyScrappy https://www.analyticsvidhya.com/blog/2022/02/web-scraping-with-pyscrappy/
Gazpacho https://github.com/maxhumber/gazpacho
ScrapeSimple Website: https://www.scrapesimple.com
Content Grabber https://contentgrabber.com/Manual/understanding_the_concept.htm
Crawly https://crawly.diffbot.com/
Apify https://apify.com/
Mozenda Website: https://www.mozenda.com/
obsei https://github.com/lalitpagaria/obsei
Diffbot https://analyticsindiamag.com/diffbot/
Trustpilot,webhose,scrapingbot
lxml https://lxml.de/index.html#introduction
ScrapingBee https://analyticsindiamag.com/scrapingbee-api/
Scrape HTML tables https://www.youtube.com/watch?v=6U5xJ3mXRKA&feature=youtu.be or pd.read_html
requests-html https://github.com/kennethreitz/requests-html
newspaper https://github.com/codelucas/newspaper https://www.youtube.com/watch?v=Hfry5XnISyc
newspaper3k: https://newspaper.readthedocs.io # easily extract text from articles
newscatcher https://github.com/kotartemiy/newscatcher https://www.youtube.com/watch?v=pHzOuizZq4I
patang (extract product details) https://github.com/tejazz/patang
lisc https://github.com/lisc-tools/lisc
Helena WEB AUTOMATION FOR END USERS https://helena-lang.org/
pandas(read_html)
wget,curl,parsehub,webhouse,octoparse,scraping bot,scraping bee,Common,Content Grabber,Docparser,Scraper API,Import.io,Altair Monarch,WebAutomation.io,WebScraper.io,Scrape.do, AvesAPI, ParseHub, Import.io, Octoparse, Scrapingdog, Diffbot, ScrapingBee, Grepsr, Scraper API, Scrapy
Crawl Crawly https://crawly.diffbot.com/
HTML basics for web scraping,Web Scraping with Octoparse,Web Scraping with Selenium
10-best-web-scraping-tools https://www.scraperapi.com/blog/the-10-best-web-scraping-tools/
https://www.kdnuggets.com/2021/02/6-web-scraping-tools.html
https://analyticsindiamag.com/complete-learning-path-to-web-scraping-with-all-major-tools/ https://towardsdatascience.com/6-web-scraping-tools-that-make-collecting-data-a-breeze-457c44e4411d
https://towardsdatascience.com/6-web-scraping-tools-that-make-collecting-data-a-breeze-457c44e4411d https://www.kdnuggets.com/2018/02/web-scraping-tutorial-python.html
https://www.octoparse.com/ https://github.com/tirthajyoti/pydbgen https://www.mozenda.com/ https://www.mockaroo.com/ https://lionbridge.ai/ https://www.mturk.com/ https://appen.com/
11.GoogleImageCrawler,google_images_download,bing_image
https://www.freepik.com/popular-photos , https://stocksnap.io/ , https://www.pexels.com/ ,https://unsplash.com/ , https://pixabay.com/
b.Web Crawling
https://python.libhunt.com/scrapy-alternatives
Flat Data https://octo.github.com/projects/flat-data
b.3rd party API'S
22 APIs every data scientist should learn https://www.springboard.com/library/data-science/top-apis-for-data-scientists/
c.creating own data (manual collection eg:google docx,servey,etc...) primary data
d.etl awesome ETL https://github.com/pawl/awesome-etl#python https://github.com/achuthasubhash/awesome-etl
38x faster data pipelines with tf.data
d.Databases
Databases are 2 kind sequel and no sequel database
sql,sql lite,mysql,mongodb,montydb,hadoop,elastic search,cassendra,amazon s3,hive,googlebigtable,AWS DynamoDB,HBase,oracle db
sql https://mode.com/sql-tutorial/ https://www.w3schools.com/sql/
sql in python https://medium.com/jbennetcodes/how-to-rewrite-your-sql-queries-in-pandas-and-more-149d341fc53e
PyMongo https://analyticsindiamag.com/guide-to-pymongo-a-python-wrapper-for-mongodb/
Cloud AI Data labeling service https://cloud.google.com/ai-platform/data-labeling/docs?utm_source=youtube&utm_medium=Unpaidsocial&utm_campaign=guo-20200503-Data-Labeling
e.Online resources - ultimate resource https://datasetsearch.research.google.com/ https://medium.com/swlh/where-to-find-awesome-machine-learning-datasets-6bb909a3f350
10 BEST DATA COLLECTION TOOLS FOR EFFECTIVE RESULTS https://www.analyticsinsight.net/10-best-data-collection-tools-for-effective-results/
https://www.freecodecamp.org/news/https-medium-freecodecamp-org-best-free-open-data-sources-anyone-can-use-a65b514b0f2d/ https://research.google/tools/datasets/
Machine learning datasets https://www.datasetlist.com/ https://wiki.pathmind.com/open-datasets
https://guides.library.cmu.edu/az.php https://docs.microsoft.com/en-us/azure/azure-sql/public-data-sets https://registry.opendata.aws/ https://paperswithcode.com/datasets https://datasets.quantumstat.com/ https://www.quandl.com/ http://dataportals.org/ https://opendatamonitor.eu/frontend/web/index.php?r=dashboard%2Findex https://en.wikipedia.org/wiki/List_of_datasets_for_machine-learning_research https://www.quora.com/Where-can-I-find-large-datasets-open-to-the-public https://www.reddit.com/r/datasets/ https://ourworldindata.org/ https://data.worldbank.org/ https://data.world/ https://data.census.gov/cedsci/ https://data.seattle.gov/ https://www.openml.org/ https://visualdata.io/discovery
World’s Largest Data Platform https://worlddata.ai/
Awesome list of datasets in 100+ categories https://www.kdnuggets.com/2021/05/awesome-list-datasets.html
https://sebastianraschka.com/blog/2021/ml-dl-datasets.html https://enoumen.com/2021/04/23/data-sciences-datasets-data-visualization-data-analytics-big-data-data-lakes/
https://serokell.io/blog/best-machine-learning-datasets https://medium.com/@ODSC/25-excellent-machine-learning-open-datasets-940ca2124dfc
1)kaggle-https://www.kaggle.com/datasets , 𝚙𝚒𝚙 𝚒𝚗𝚜𝚝𝚊𝚕𝚕 𝚔𝚊𝚐𝚐𝚕𝚎𝚍𝚊𝚝𝚊𝚜𝚎𝚝𝚜
Downloading Kaggle datasets directly into Google Colab -https://towardsdatascience.com/downloading-kaggle-datasets-directly-into-google-colab-c8f0f407d73a
How to Download Kaggle Datasets using Jupyter Notebook https://www.analyticsvidhya.com/blog/2021/04/how-to-download-kaggle-datasets-using-jupyter-notebook/
2)https://sebastianraschka.com/blog/2021/ml-dl-datasets.html
movielens-https://grouplens.org/datasets/movielens/latest/
dagshub datset https://dagshub.com/explore/datasets
100+ of the Best Free Data Sources For Your Next Project https://www.columnfivemedia.com/100-best-free-data-sources-infographic/
World and national data, maps & rankings https://knoema.com/atlas/sources
3)data.gov-https://data.gov.in/
4)uci-https://archive.ics.uci.edu/ml/datasets.php https://github.com/tirthajyoti/UCI-ML-API
5)Group Lens dataset https://grouplens.org/
Wikipedia ML Datasets https://en.wikipedia.org/wiki/List_of_datasets_for_machine-learning_research
AWS Open Data Registry,data.gov (portals),YELP Open dataset,UNICEF Dataset,Big Bad NLP Database,Microsoft Dataset
6)world3bank https://data.world/ , worldbank
7)Google Cloud BigQuery public datasets
Google Public Datasets-cloud.google.com/bigquery/public-data/
Google Cloud Data Catalog https://cloud.google.com/data-catalog
Academic Torrents-https://academictorrents.com/check.htm?returnto=%2Fbrowse.php
8)online hacktons
Datasets https://www.paperswithcode.com/datasets
9)image data from google_images_download
https://www.visualdata.io/discovery
http://xviewdataset.org/#dataset
https://ai.googleblog.com/2016/09/introducing-open-images-dataset.html
10)image data from Bing_Search
image data from simple_image_download https://github.com/RiddlerQ/simple_image_download
11)https://www.columnfivemedia.com/100-best-free-data-sources-infographic
graviti Unleash the Power of Unstructured Data https://www.graviti.com/?utm_medium=0730Ismael
12)Reddit:https://lnkd.in/dv5UCD4 https://www.reddit.com/r/datasets/
praw.Reddit https://github.com/praw-dev/praw
13)https://datasets.bifrost.ai/?ref=producthunt
14)data.world:https://lnkd.in/gEK897K
15)https://data.world/datasets/open-data
https://tinyletter.com/data-is-plural
16)FiveThirtyEight :- https://lnkd.in/gyh-HDj , https://data.fivethirtyeight.com/
17)BuzzFeed :- https://lnkd.in/gzPWyHj
Buzzfeed News -github.com/BuzzFeedNews
Socrata - https://opendata.socrata.com/
18)Google public datasets :- https://lnkd.in/g5dH8qE
Statistics Canada https://www.statcan.gc.ca/eng/start https://towardsdatascience.com/how-to-collect-data-from-statistics-canada-using-python-db8a81ce6475
Deep Image Search AI-based image search engine https://github.com/TechyNilesh/DeepImageSearch
https://www.datasciencecentral.com/profiles/blogs/big-data-sets-available-for-free
19)Quandl :- https://www.quandl.com stock data
statista : https://www.statista.com/ stock data
20)socorateopendata :- https://lnkd.in/gea7JMz
21)AcedemicTorrents :- https://lnkd.in/g-Ur9Xy
22) Automates Image Annotation for Deep Learning Models https://medium.com/towards-artificial-intelligence/improving-data-labeling-efficiency-with-auto-labeling-uncertainty-estimates-and-active-learning-5848272365be
Label Studio,Sloth,LabelBox,TagTog,Amazon SageMaker GroundTruth,Playment,Superannotate,Playment,Dataturk,LightTag,Superannotate,CVAT,sloth,LabelImg,cvat
Automate data preparation https://www.superb-ai.com/
https://neptune.ai/blog/annotation-tool-comparison-deep-learning-data-annotation?utm_source=linkedin&utm_medium=post&utm_campaign=blog-annotation-tool-comparison-deep-learning-data-annotation
Diffgram,Label Studio ,CVAT,SuperAnnotate,Datasaur https://anthony-sarkis.medium.com/the-5-best-ai-data-annotation-platforms-for-machine-learning-2021-ec17c15142f3
https://foobar167.medium.com/open-source-free-software-for-image-segmentation-and-labeling-4b0332049878
***Label Assist: Model Assisted Pre-Annotation for Computer Vision https://blog.roboflow.com/announcing-label-assist/ https://www.youtube.com/watch?v=919CihTlkZw&feature=youtu.be***
https://github.com/jsbroks/awesome-dataset-tools
makeml https://makeml.app/
superannotate https://www.superannotate.com/
jupyter-innotater data annotator for Jupyter notebooks https://github.com/ideonate/jupyter-innotater
JupyterLab extension for annotating data https://github.com/explosion/jupyterlab-prodigy
semi-auto-image-annotation-tool https://github.com/virajmavani/semi-auto-image-annotation-tool
labelimage:- https://github.com/wkentaro/labelme , https://github.com/tzutalin/labelImg
labelCloud lightweight tool for labeling 3D bounding boxes in point clouds https://github.com/ch-sa/labelCloud
labeller https://www.labellerr.com/
prodigy Radically efficient machine teaching An annotation tool powered by active learning https://prodi.gy/
Labelbox-https://labelbox.com/
Playment-https://playment.io/
SuperAnnotate -https://www.superannotate.com/
CVAT-https://github.com/openvinotoolkit/cvat
Lionbridge- https://lionbridge.ai/
LinkedAI: A No-code Data Annotations- https://analyticsindiamag.com/linkedai/
Dataturks
V7 Darwin The Rapid Image Annotator https://docs.v7labs.com/docs/loading-a-dataset-in-python https://github.com/v7labs/darwin-py#usage-as-a-python-library
https://waliamrinal.medium.com/top-and-easy-to-use-open-source-image-labelling-tools-for-machine-learning-projects-ffd9d5af4a20
https://github.com/heartexlabs/awesome-data-labeling
Label a Dataset with a Few Lines of Code https://eric-landau.medium.com/label-a-dataset-with-a-few-lines-of-code-45c140ff119d
https://analyticsindiamag.com/complete-guide-to-data-labelling-tools/ https://neptune.ai/blog/data-labeling-software
Extraction of Objects In Images and Videos Using 5 Lines of Code https://towardsdatascience.com/extraction-of-objects-in-images-and-videos-using-5-lines-of-code-6a9e35677a31
https://neptune.ai/blog/data-labeling-software?utm_source=twitter&utm_medium=tweet&utm_campaign=blog-data-labeling-software
23)tensorflow_datasets as tfds https://www.tensorflow.org/datasets (import tensorflow_datasets as tfds)
https://lionbridge.ai/datasets/tensorflow-datasets-machine-learning/
24)https://datasets.bifrost.ai/?ref=producthunt
25)https://ourworldindata.org/
26)https://data.worldbank.org/
27)google open images:https://storage.googleapis.com/openimages/web/download.html
30 Largest TensorFlow Datasets for Machine Learning https://lionbridge.ai/datasets/tensorflow-datasets-machine-learning/
https://cloud.google.com/bigquery/public-data/ https://towardsdatascience.com/bigquery-public-datasets-936e1c50e6bc
https://christopherzita.medium.com/how-to-download-google-images-using-python-2021-82e69c637d59
28)https://data.gov.in/
29)imagenet dataset-http://www.image-net.org/
30)https://parulpandey.com/2020/08/09/getting-datasets-for-data-analysis-tasks%e2%80%8a-%e2%80%8aadvanced-google-search/
31)https://storage.googleapis.com/openimages/web/index.html ,
https://storage.googleapis.com/openimages/web/visualizer/index.html?set=train&type=segmentation&r=false&c=%2Fm%2F09qck
https://console.cloud.google.com/marketplace/browse?filter=solution-type:dataset&_ga=2.35328417.1459465882.1589693499-869920574.1589693499
https://catalog.data.gov/dataset?groups=education2168#topic=education_navigation
https://vincentarelbundock.github.io/Rdatasets/datasets.html
32)coco dataset https://cocodataset.org/#explore
33)huggingface datasets-https://github.com/huggingface/datasets https://huggingface.co/datasets https://huggingface.co/languages
pip install datasets
34)Big Bad NLP Database-https://datasets.quantumstat.com/
fast.ai Datasets https://course.fast.ai/datasets
https://github.com/niderhoff/nlp-datasets
600 NLP Datasets and Glory https://pub.towardsai.net/600-nlp-datasets-and-glory-4b0080bf5ab
nlp-datasets https://github.com/karthikncode/nlp-datasets
https://analyticsindiamag.com/15-most-important-nlp-datasets/ https://medium.com/ai-in-plain-english/25-free-datasets-for-natural-language-processing-57e407402c60
35)https://www.edureka.co/blog/25-best-free-datasets-machine-learning/
36)bigquery public dataset ,Google Public Data Explorer
https://cloud.google.com/public-datasets https://guides.library.cmu.edu/machine-learning/datasets
37)inbuilt library data eg:iris dataset,mnist dataset,etc...
pandas-datareader https://github.com/pydata/pandas-datareader
tf.data.Datasets for TensorFlow Datasets
38)https://data.gov.sg/ https://data.gov.au/ https://data.europa.eu/euodp/en/data https://data.europa.eu/euodp/en/data https://data.govt.nz/
data.gov.be ,data.egov.bg/ ,data.gov.cz/english ,portal.opendata.dk,govdata.de,opendata.riik.ee,data.gov.ie,data.gov.gr,datos.gob.es,data.gouv.fr,data.gov.hr
dati.gov.it,data.gov.cy,opendata.gov.lt,data.gov.lv,data.public.lu,data.gov.mt,data.overheid.nl,data.gv.at,danepubliczne.gov.pl,dados.gov.pt,data.gov.ro,podatki.gov.si
data.gov.sk,avoindata.fi,oppnadata.se,https://data.adb.org/ ,https://data.iadb.org/ ,https://www.weforum.org/agenda/2018/03/latin-america-smart-cities-big-data/
https://data.fivethirtyeight.com/ , https://wiki.dbpedia.org/ ,https://www.europeandataportal.eu/en ,https://data.europa.eu/ ,https://www.census.gov/,
https://www.who.int/data/gho ,https://data.unicef.org/open-data/ ,http://data.un.org/ ,https://data.oecd.org/ ,https://data.worldbank.org/
39.Awesome Public Dataset- https://github.com/awesomedata/awesome-public-datasets
Get OpenML’s Dataset in One Line of Code https://mathdatasimplified.com/2021/04/23/fetch_openml-get-openmls-dataset-in-one-line-of-code/
https://github.com/the-pudding/data
datasets https://github.com/benedekrozemberczki/datasets
kdnuggets https://www.kdnuggets.com/datasets/index.html
Hub https://github.com/activeloopai/Hub
40.Datasets for Machine Learning on Graphs-https://ogb.stanford.edu/
41.https://www.johnsnowlabs.com/data/
42.30 largest tensorflow datasets-https://lionbridge.ai/datasets/tensorflow-datasets-machine-learning/
43. coco dataset-https://cocodataset.org/#home
flickr-downloader https://github.com/renatoviolin/flickr-downloader/
Google Open images-https://opensource.google/projects/open-images-dataset https://storage.googleapis.com/openimages/web/index.html
50+ Object Detection Datasets-https://medium.com/towards-artificial-intelligence/50-object-detection-datasets-from-different-industry-domains-1a53342ae13d
70+ Image Classification Datasets from different Industry domains-https://medium.com/towards-artificial-intelligence/70-image-classification-datasets-from-different-industry-domains-part-2-cd1af6e48eda
VisualData Discovery https://www.visualdata.io/discovery https://guides.library.cmu.edu/machine-learning/datasets
data https://storage.googleapis.com/openimages/web/visualizer/index.html?c=%2Fm%2F04yqq2&r=false&set=train&type=segmentation&utm_campaign=Weekly%20Machine%20Learning%20news&utm_medium=email&utm_source=Revue%20newsletter
VisualData https://www.visualdata.io/discovery
bifrost- https://datasets.bifrost.ai/
satellite images https://towardsdatascience.com/finding-satellite-images-for-your-data-science-project-888695361925
https://public.roboflow.com/
https://www.visualdata.io/discovery http://www.image-net.org/ https://www.cs.toronto.edu/~kriz/cifar.html
tensorflow_datasets.object_detection - https://storage.googleapis.com/openimages/web/index.html
https://github.com/google-research-datasets/Objectron/ https://ai.googleblog.com/2020/11/announcing-objectron-dataset.html?m=1
http://idd.insaan.iiit.ac.in/ http://database.mmsp-kn.de/koniq-10k-database.html
https://ai.googleblog.com/2020/11/announcing-objectron-dataset.html
https://www.visualdata.io/discovery https://blogs.bing.com/maps/2019-03/microsoft-releases-12-million-canadian-building-footprints-as-open-data
https://blogs.bing.com/maps/2019-09/microsoft-releases-18M-building-footprints-in-uganda-and-tanzania-to-enable-ai-assisted-mapping
https://datasets.bifrost.ai/ https://storage.googleapis.com/openimages/web/download.html https://computervisiononline.com/datasets http://yacvid.hayko.at/
https://www.cogitotech.com/use-cases/biodiversity/
ImageNet data -http://image-net.org/
ApolloScape Dataset-http://apolloscape.auto/
https://github.com/chrieke/awesome-satellite-imagery-datasets
44.https://github.com/fivethirtyeight/data
45.Recommender Systems Datasets-https://cseweb.ucsd.edu/~jmcauley/datasets.html
46.indiadataportal-https://indiadataportal.com/
47.US Government Open Dataset: https://www.data.gov/
https://censusreporter.org/ https://data.census.gov/cedsci/
48.AWS Public Data Sets:https://registry.opendata.aws/ https://aws.amazon.com/opendata/?wwps-cards.sort-by=item.additionalFields.sortDate&wwps-cards.sort-order=desc
49.https://the-eye.eu/public/AI/pile_preliminary_components/
Reddit -https://www.reddit.com/r/datasets/
wikipedia-https://en.wikipedia.org/wiki/List_of_datasets_for_machine-learning_research
http://opendata.cern.ch/ , https://www.imf.org/en/Data
Global Health Observatory data repository-https://apps.who.int/gho/data/node.main
CERN Open Data Portal-http://opendata.cern.ch/
TensorFlow Datasets https://www.tensorflow.org/datasets
50.openblender- https://www.openblender.io/#/welcome
51.Top 10 Datasets For Cybersecurity Projects- https://analyticsindiamag.com/top-10-datasets-for-cybersecurity-projects/
52.Datasets from Web Crawl Data (nlp)-http://data.statmt.org/cc-100/
53.https://www.springboard.com/blog/free-public-data-sets-data-science-project/
54.NASA - https://nasa.github.io/data-nasa-gov-frontpage/ace
55.Academic Torrents,GitHub Datasets,CERN Open Data Portal,Global Health Observatory Data Repository
56.32 Data Sets to Uplift your Skills in Data Science-https://blog.datasciencedojo.com/data-sets-data-science-skills/?utm_content=144243072&utm_medium=social&utm_source=linkedin&hss_channel=lcp-3740012
https://lionbridge.ai/datasets/the-50-best-free-datasets-for-machine-learning/
57.OpenDaL-https://opendatalibrary.com/
Data Is Plural-https://docs.google.com/spreadsheets/d/1wZhPLMCHKJvwOkP4juclhjFgqIY8fQFMemwKL2c64vk/edit#gid=0
VisualData-https://www.visualdata.io/discovery
https://medium.com/towards-artificial-intelligence/best-datasets-for-machine-learning-data-science-computer-vision-nlp-ai-c9541058cf4f
58.Pandas Data Reader-https://pandas-datareader.readthedocs.io/en/latest/remote_data.html
59.ieee-dataport-https://ieee-dataport.org/datasets
https://medium.com/towards-artificial-intelligence/best-datasets-for-machine-learning-data-science-computer-vision-nlp-ai-c9541058cf4f
https://github.com/neomatrix369/awesome-ai-ml-dl/blob/master/data/datasets.md#datasets-and-sources-of-raw-data
60.Generating Realistic Fake Data https://towardsdatascience.com/free-resources-for-generating-realistic-fake-data-da63836be1a8
Full Synthetic Data ,Partial Synthetic Data,Hybrid Synthetic Data
Faker is a Python package that generates fake data-https://github.com/joke2k/faker
ydata-synthetic,Gretel,gretel-synthetics,GenerateData,DataSynthesizer,SDV,SDGym,SDMetrics,Copulas,gretel-synthetics,kubric,CTGAN,Synthea,synthia,nbsynthetic ,pydbgen,synthpop,faker,Tonic,ydata,Mostly AI,Mirry.ai,Hazy,Gretel,Diveplane,Datagen,Mimesis,faker,FauxFactory,Radar,PikaAccelario,Chooch,Datagen,Datomize,Deep Vision Data,Monitaur,MOSTLY AI,OpenSynthetics,Replica Analytics,Scale AI,SKY ENGINE AI,Synthesis AI,Plaitpy,TimeseriesGenerat,Accelario,Chooch,dgutils,AI.Reverie,Kinetic Vision,SynthDet,OpenSynthetics,Mockaroo,GenerateData,JSON Schema Faker,FakeStoreAPI,Mock Turtle,nbsynthetic,AiFi,AI.Reverie,Anyverse,Cvedia,DataGen,Diveplane,Gretel,Hazy,Mostly AI,OneView,TRGD,YDATA Synthetic,SDV,Tonic.AI,Mostly.AI,Parallel Domain,Mindtech,Synthesis AI,Oneview,Hazy,CVEDIA,SKY ENGINE AI,Edgecase.ai,Statice,ANYVERSE,Rendered.ai,Datomize,Facteus,Gretel,Synthesized,Syntheticus,Syntho,Tonic, kubric,Stable Diffusion,Parallel Domain,Mindtech,Synthesis AI,Oneview,MOSTLY AI,Hazy,CVEDIA,SKY ENGINE AI,Edgecase.ai,Statice,ANYVERSE,Rendered.ai,Datomize,Facteus,Gretel,Synthesized,Syntheticus,Syntho,Tonic,MOSTLY AI, GenRocket, YData, Hazy, and MDClone ,Gretel, MOSTLY AI, Hazy, Statice ,NVIDIA Omniverse, OneView, CVEDIA, Datagen, Parallel Domain,Infinity AI,Parallel Domain,Rendered.AI,Scale.AI,SKY ENGINE AI,Synthesis AI,Paella,statice,DataSynthesizer,Pydbgen,TimeseriesGenerator,Mimesis,Synthesized,Syntheticus,Syntho,Tonic,Clearbox AI ,RDT (Reversible Data Transforms),DeepEcho
Models: GANs, CTGAN, WGAN, WGAN-GP, VAEs,GANs, TimeGAN, AR
GAN-based Deep Learning data synthesizer CTGAN,CopulaGAN,Synthetic Data Vault,Probabilistic AutoRegressive model
Extract the metadata using DataDescriber, Compare the input and synthetic data using ModelInspector
Mockaroo https://www.mockaroo.com/
GenerateData https://site.generatedata4.com/
JSON Schema Faker https://json-schema-faker.js.org/
FakeStoreAPI https://fakestoreapi.com/
graviti dataset https://gas.graviti.com/open-datasets
Synthetic data for computer vision https://github.com/ZumoLabs/zpy
GANs for Tabular Synthetic Data Generation https://github.com/Diyago/GAN-for-tabular-data
Synthetic Image Datasets https://analyticsindiamag.com/unity-launches-synthetic-image-datasets-to-train-ai-models-faster/
Synthetic structured data generators https://github.com/ydataai/ydata-synthetic
gretel Synthetic Data API https://gretel.ai/
Timeseries DGAN https://synthetics.docs.gretel.ai/en/latest/models/timeseries_dgan.html
DatasetGAN: an automatic procedure to generate massive datasets of high-quality images
Generating synthetic tabular data with GANs,Synthetic Time-Series Data by A GAN approach
Unity Launches Synthetic Image Datasets https://www.marktechpost.com/2021/04/23/unity-launches-synthetic-image-datasets-to-train-ai-and-computer-vision-models-faster/
Generate Your Own Dataset using GAN https://www.analyticsvidhya.com/blog/2021/04/generate-your-own-dataset-using-gan/
accurate of synthetic data https://gretel.ai/blog/how-accurate-is-my-synthetic-data
Synthetic data library https://github.com/finos/datahub https://github.com/agmmnn/awesome-blender https://opendata.blender.org/ https://www.youtube.com/watch?v=eZwOeBkLL8E
https://www.kdnuggets.com/2019/09/scikit-learn-synthetic-dataset.html
Fully Synthetic Data,Partially Synthetic Data ,Hybrid Synthetic Data https://towardsdatascience.com/synthetic-data-key-benefits-types-generation-methods-and-challenges-11b0ad304b55
Synthetic Image Datasets https://analyticsindiamag.com/unity-launches-synthetic-image-datasets-to-train-ai-models-faster/ https://dockship.io/articles/607847e461373d1b994cc2dc/create-synthetic-images-using-opencv-(python)
gretel-synthetics Synthetic data generators for structured and unstructured text, featuring differentially private learning. https://github.com/gretelai/gretel-synthetics
Synthetic Data Generation Using Gaussian Mixture Model https://deepnote.com/@chanakya-vivek-kapoor/Synthetic-Data-Generation-QaaTRs73T2iCb0amHFbwpQ
Synthetic Data Vault https://analyticsindiamag.com/guide-to-synthetic-data-vault-an-ecosystem-of-synthetic-data-generation-libraries/ https://github.com/sdv-dev/SDV
Create Your own Image Dataset using Opencv https://www.analyticsvidhya.com/blog/2021/05/create-your-own-image-dataset-using-opencv-in-machine-learning/
ydata-synthetic https://github.com/ydataai/ydata-synthetic
Table Evaluator About Evaluate real and synthetic datasets with each other https://github.com/Baukebrenninkmeijer/table-evaluator
evaluate quality and efficacy of synthetic datasets SDMetrics https://github.com/sdv-dev/SDMetrics
61.Text Data Annotator Tool - Datasaur https://datasaur.ai/
Tagalog is our state-of-the-art solution for data management and labeling in Natural Language Processing https://www.tagalog.ai/tagalog/
62.Google Analytics cost data import https://segmentstream.com/google-analytics?utm_source=twitter&utm_medium=cpc&utm_campaign=ga_costs_import_en&utm_content=guide
63.https://lionbridge.ai/services/crowdsourcing/ https://lionbridge.ai/ https://www.clickworker.com/ https://appen.com/ https://www.globalme.net/
64.Azure Open Datasets https://azure.microsoft.com/en-us/services/open-datasets/ https://azure.microsoft.com/en-in/services/open-datasets/catalog/
Yelp Open Dataset https://www.yelp.com/dataset
https://data.world/
ODK Open Data Kit- https://getodk.org/
World Bank Open Data https://data.worldbank.org/
https://analyticsindiamag.com/10-biggest-data-breaches-that-made-headlines-in-2020/
https://data.mendeley.com/
https://github.com/iamtekson/geospatial-data-download-sites
https://eugeneyan.com/writing/data-discovery-platforms/
65.https://medium.com/towards-artificial-intelligence/best-datasets-for-machine-learning-data-science-computer-vision-nlp-ai-c9541058cf4f
https://towardsdatascience.com/data-repositories-for-almost-every-type-of-data-science-project-7aa2f98128b
https://github.com/MTG/freesound-datasets
https://dataform.co/
https://github.com/rfordatascience/tidytuesday https://www.youtube.com/watch?v=vCBeGLpvoYM
https://www.analyticsvidhya.com/blog/2020/12/top-15-datasets-of-2020-that-every-data-scientist-should-add-to-their-portfolio/?utm_source=linkedin&utm_medium=AV|link|high-performance-blog|blogs|44181|0.375
https://cseweb.ucsd.edu/~jmcauley/datasets.html
66.https://en.wikipedia.org/wiki/List_of_datasets_for_machine-learning_research
https://archive.org/details/datasets
https://commoncrawl.org/
https://www.youtube.com/watch?v=1aUt8zAG09E
67. 6 Sources of Financial Data https://medium.datadriveninvestor.com/financial-data-431b75975bb
yfinance for finance data using https://github.com/ranaroussi/yfinance https://medium.com/towards-artificial-intelligence/algorithmic-trading-with-python-and-machine-learning-part-1-47c56706c182
import fix_yahoo_finance as yf , yahoofinancials ,Pandas DataReaders,Twelve Data
financeapi https://towardsdatascience.com/pull-and-analyze-financial-data-using-a-simple-python-package-83e47759c4a7
Investing.com pip install investpy ,Kite by Zerodha pip install kiteconnect,quandl pip install quandl
https://www.analyticsvidhya.com/blog/2021/01/bear-run-or-bull-run-can-reinforcement-learning-help-in-automated-trading/?utm_source=feedburner&utm_medium=email&utm_campaign=Feed%3A+AnalyticsVidhya+%28Analytics+Vidhya%29
Downloading Historical Stock prices with Alpha Vantage https://medium.com/towards-artificial-intelligence/downloading-historical-stock-prices-with-alpha-vantage-688edad46a6d
Pandas Datareader https://pandas-datareader.readthedocs.io/en/latest/ https://www.youtube.com/watch?v=f2BCmQBCwDs
Get Financial Data Directly into Python https://www.quandl.com/tools/python https://medium.com/nerd-for-tech/how-to-get-financial-data-using-python-7a508f25fc39
openml https://www.openml.org/search?type=data
https://registry.opendata.aws/
voice_datasets https://github.com/jim-schwoebel/voice_datasets
Dynamically-Generated-Hate-Speech-Dataset https://github.com/bvidgen/Dynamically-Generated-Hate-Speech-Dataset
68.DOCANO, an open source text annotation tool https://github.com/doccano/doccano
69.https://www.dataquest.io/blog/free-datasets-for-projects/
70.audio set https://research.google.com/audioset/
71.FlatData Flat explores how to make it easy to work with data in git and GitHub https://octo.github.com/projects/flat-data?utm_campaign=Data_Elixir&utm_source=Data_Elixir_337
72.Snorkel is an open-source Python library for programmatically building training datasets without manual labeling. https://www.snorkel.org/ https://towardsdatascience.com/snorkel-programmatically-build-training-data-in-python-712fc39649fe
2.Feature engineering
Feature-engine https://trainindata.medium.com/feature-engine-a-new-open-source-python-package-for-feature-engineering-29a0ab88ea7c https://feature-engine.readthedocs.io/en/latest/ https://github.com/solegalli/feature_engine https://www.datasciencecentral.com/profiles/blogs/feature-engine-python-package-for-feature-engineering
Automated feature engineering https://medium.com/ibm-data-ai/automated-feature-engineering-for-relational-data-with-autoai-3612fafe9f89
Automated Data Wrangling https://catalyst.coop/2021/05/23/automated-data-wrangling/
Automatic Feature Engineering Using Featurewiz https://towardsdatascience.com/automate-your-feature-selection-workflow-in-one-line-of-python-code-3d4f23b7e2c4 https://github.com/AutoViML/featurewiz
Automatic Feature Engineering Using AutoFeat https://analyticsindiamag.com/guide-to-automatic-feature-engineering-using-autofeat/
Upgini accuracy improving features https://github.com/upgini/upgini https://upgini.com/
Categorical Encoding https://github.com/scikit-learn-contrib/category_encoders
lazytransform https://github.com/AutoViML/lazytransform
Streamlining Feature Engineering Pipelines with Feature-engine https://towardsdatascience.com/streamlining-feature-engineering-pipelines-with-feature-engine-e781d551f470 https://feature-engine.readthedocs.io/en/latest/#
Validate your Data (Schema) https://towardsdatascience.com/introduction-to-schema-a-python-libary-to-validate-your-data-c6d99e06d56a
Validate Your pandas DataFrame with Pandera https://github.com/pandera-dev/pandera
Statistical DataFrame Testing Toolkit https://pandera.readthedocs.io/en/stable/index.html
Data storing format:Pickle,Parquet,Feather,Avro,ORC
Data cleaning-Pyjanitor-https://analyticsindiamag.com/beginners-guide-to-pyjanitor-a-python-tool-for-data-cleaning/
data cleaning library https://www.analyticsvidhya.com/blog/2021/05/data-cleaning-libraries-in-python-a-gentle-introduction/
Mage https://github.com/mage-ai/mage-ai
Cleaner Data Analysis with Pandas Using Pipes https://towardsdatascience.com/cleaner-data-analysis-with-pandas-using-pipes-4d73770fbf3c
DataPrep https://dataprep.ai/ https://github.com/sfu-db/dataprep https://towardsdatascience.com/dataprep-v0-3-0-has-been-released-be49b1be0e72
Dora (pip library) - data cleaning
Dora,PrettyPandas,DataCleaner,Tabulate,Pyjanitor,OpenRefine,cleanlab,pandera
https://github.com/sfu-db/dataprep https://github.com/akanz1/klib https://www.bitrook.com/ https://github.com/rhiever/datacleaner https://github.com/johnkerl/miller
cleanlab data-centric AI and machine learning with label errors, finding mislabeled data, and uncertainty quantification. Works with most datasets and models https://github.com/cleanlab/cleanlab
cleantext https://www.youtube.com/watch?v=i2TjAgga1YU&feature=youtu.be
CleanText: A Python Package to Clean Raw Text Data https://analyticsindiamag.com/guide-to-cleantext-a-python-package-to-clean-raw-text-data/
ATOM https://github.com/tvdboom/ATOM https://towardsdatascience.com/how-to-test-multiple-machine-learning-pipelines-with-just-a-few-lines-of-python-1a16cb4686d
openrefine A free, open source, powerful tool for working with messy data https://openrefine.org/#
data leaning library https://www.analyticsvidhya.com/blog/2021/05/data-cleaning-libraries-in-python-a-gentle-introduction/
https://machinelearningmastery.com/basic-data-cleaning-for-machine-learning/
Speed Up Data Cleaning and Exploratory Data Analysis in Python with klib https://github.com/akanz1/klib https://towardsdatascience.com/speed-up-your-data-cleaning-and-preprocessing-with-klib-97191d320f80
missingno https://github.com/ResidentMario/missingno
Take the Pain Out of Data Cleaning for Machine Learning https://towardsdatascience.com/take-the-pain-out-of-data-cleaning-for-machine-learning-20a646a277fd
dabl https://ms-bharti.medium.com/jump-start-your-supervised-learning-task-with-dabl-e479323e81fe
Easy to use Python library of customized functions for cleaning and analyzing data https://github.com/akanz1/klib
PyOD https://pyod.readthedocs.io/en/latest/ https://github.com/yzhao062/pyod/blob/development/docs/index.rst https://towardsdatascience.com/how-to-detect-outliers-with-python-pyod-aa7147359e4b
Amazon’s New Visual Data Cleaning Tool Can Speed Up Your AI Projects https://medium.com/dataseries/how-amazons-new-visual-data-tool-can-speed-up-your-ai-projects-68e3289382c
Featuretools https://www.featuretools.com/ https://towardsdatascience.com/why-automated-feature-engineering-will-change-the-way-you-do-machine-learning-5c15bf188b96
https://github.com/alteryx/featuretools https://analyticsindiamag.com/introduction-to-featuretools-a-python-framework-for-automated-feature-engineering/
Feature Selection using Genetic Algorithm https://github.com/kaushalshetty/FeatureSelectionGA
AutoFeat https://analyticsindiamag.com/guide-to-automatic-feature-engineering-using-autofeat/ https://github.com/cod3licious/autofeat
feast Feature Store for Machine Learning https://github.com/feast-dev/feast https://www.youtube.com/watch?v=ZeJdr0nZ9PA
Category Encoders https://contrib.scikit-learn.org/category_encoders/
Feature-engine https://feature-engine.readthedocs.io/en/latest/index.html
FeatureTools,AutoFeat,TsFresh,Cognito,OneBM,ExploreKit,PyFeat,Category Encoders,Feature-engine
Automated Feature Selection: Featurewiz https://github.com/AutoViML/featurewiz https://towardsdatascience.com/featurewiz-fast-way-to-select-the-best-features-in-a-data-9c861178602e
zoofs a Python library for performing feature selection https://github.com/jaswinder9051998/zoofs
Feature Engineering of DateTime Variables for Data Science, Machine Learning https://www.kdnuggets.com/2021/04/feature-engineering-datetime-variables-data-science-machine-learning.html
NeatText a simple NLP package for cleaning textual data and text preprocessing https://github.com/Jcharis/neattext
Remove duplicate data in dataset,Data validity check,Contaminated Data,Inconsistent Data,Invalid Data,
Feature Selection
1.Removal of arbitraty features: DropFeatures
Removing unused columns,Removing Constant features,Removing Constant Features using VarianceThreshold,Removing Quasi-Constant Features,Removing Duplicate Columns
2.Removal of constant and almost constant features: DropConstantFeatures
Removal of Low Variance
removal of irrelevant data
3.Removal of duplicated variables: DropDuplicateFeatures
4.Removal of correlated features: DropCorrelatedFeatures, SmartCorrelatedSelection
Drop features that have a poor correlation with the response variable
5.Selection of features by value shuffling: SelectByShuffling
Selection of features by High correlation with the target variable
6.Selection of features by univariate performance: SelectBySingleFeaturePerformance
7.Selection of features by target encoding: SelectByTargetMeanPerformance
8.Recursive Feature Elimination: RecursiveFeatureElimination
9.Recursive Feature Addition: RecursiveFeatureAddition
stats,Scipy,Pingouin,Statsmodels,SymPy,Sage,
StatisticsGen component computes statistics
Check data types , Handle duplicate values
a.Handle missing value
Types of missing value https://datamuni.com/@atsunorifujita/missing-value-imputation-using-datawig
Handling Missing Values in Pandas https://pub.towardsai.net/handling-missing-values-in-pandas-f87cec928937
Identify the source of missing data
i.missing completely at random(no correlation b/w missing and observed data) we can delete no disturbance of data distribution
ii.missing at random (randomness in missing data, missing value have correlation by data) we can't delete because disturbance of data distribution
iii.missing not at random (there is reason for missing value and directly related to value)
iv.structured missing 100 % sure on why it is missing
Identify Missingness Types With Missingno https://towardsdev.com/how-to-identify-missingness-types-with-missingno-61cfe0449ad9
Univariate,Multivariate https://medium.com/fintechexplained/what-are-imputers-in-data-science-b72f8308322b
univariate imputation impute on 1 column multi variate imputation impute on 1 or more column
1.if missing data too small then delete it a.row deletion b.column deletion c.pairwise deletion and listwise deletion
Drop based on a threshold value,Drop using a subset of columns
2.replace by statistical method mean(influenced by outiler),median(not influenced by outiler),mode , minimum, maximum,Zero,Constant
Fill with Mean / Median of Column or Group Forward Fill or Forward Fill within Groups
Mean and Median Fill with Groupby
Pass another DataFrame to fillna function to fill up the missing values.
Similar case Imputation
3.apply classifier algorithm to predict missing value
Using Algorithms that support missing values
Imputation using Deep Learning Library — Datawig https://github.com/awslabs/datawig
4.Simple Imputer,and Multiple Imputation ,Iterative imputer,knn imputer, multivariate imputation, Verstack — NaNImputer,Impyute —MICE ,Substitution
5.apply unsupervised
6.Random Imputation,Iterative Imputation,Random Sample imputation
7.Adding a variable to capture NAN(missing term),Imputation with the string ‘Missing’,Adding missing idicator
8.Arbitrary Value Imputation
TREAT MISSING VALUES AS A SEPARATE CATEGORY
ue DOMAIN KNOWLEDGE
9.hot deck Imputation,Cold deck imputation
10.regression Imputation,Stochastic Regression Imputation,Interpolation and Extrapolation
11.End of Distribution Imputation
12.Arbitrary Value Imputation
13.Frequent Category Imputation
14.MICE Imputation,miceforest ( https://github.com/AnotherSamWilson/miceforest )
Miss Forest https://github.com/stekhoven/missForest
15.interpolation https://www.analyticsvidhya.com/blog/2021/06/power-of-interpolation-in-python-to-fill-missing-values/ Interpolate or Interpolate within Groups
LINEARINTERPOLATION ,POLYNOMIALINTERPOLATION,INTERPOLATION THROUGH PADDING
Extrapolation and Interpolation ,Time-Based Interpolation,Spline Interpolation,Linear Interpolation,Smoothing, interpolation,Bidirectional Recurrent Imputation for Time Series (
16.Last Observation Carried Forward (LOCF) , Next Observation Carried Backward , Rolling Statistics, Interpolation
Single and Multiple Imputation,Univariate Imputation,Multivariate Imputation ,Iterative Imputer,MissForest Imputation,Stochastic Regression Imputation, Multiple Imputations, Datawig, Hot-Deck imputation, Extrapolation, Interpolation
datawig Imputation of missing values in tables https://github.com/awslabs/datawig
Imputation using K-NN,missForest,Random Forest-based Imputation,missingpy,som,Ann,mlp
Model based procedure gaussian mixture model
Imputation Using Deep Learning (Datawig),neural network for imputation,BRITS
15.autoimpute-https://github.com/kearnz/autoimpute
16.bfill / ffill Back Fill or Back Fill within Groups
17.Adding a variable to capture NAN
18.replace NAN with a new category
19.Missing indicator
After drop or imputation feature distribution should be same
https://www.kdnuggets.com/2021/05/deal-with-categorical-data-machine-learning.html
https://towardsdatascience.com/6-different-ways-to-compensate-for-missing-values-data-imputation-with-examples-6022d9ca0779
https://stefvanbuuren.name/fimd/want-the-hardcopy.html https://www.datasciencecentral.com/profiles/blogs/how-to-treat-missing-values-in-your-data-1
20.Imputation with the string ‘Missing’ ,Addition of binary missing indicators
21.Algorithms robust to missing values - LightGBM
datawig imputation https://github.com/awslabs/datawig
22.Cluster-based approach for missing value imputation Naive clustering,Column-sensitive clustering
Top Data Cleaning Tools https://www.marktechpost.com/2022/02/20/top-data-cleaning-tools-for-data-science-and-machine-learning-projects-in-2022/
OpenRefine https://openrefine.org/ https://github.com/OpenRefine/OpenRefine
Data Ladder https://dataladder.com/
re-data fix data issues https://github.com/re-data/re-data
Automatically find and fix errors in your ML datasets. https://github.com/cleanlab/cleanlab
Clean APIs for data cleaning https://github.com/pyjanitor-devs/pyjanitor
datacleaner https://github.com/rhiever/datacleaner
https://github.com/akanz1/klib https://pyjanitor-devs.github.io/pyjanitor/ https://dataprep.ai/ https://scrubadub.readthedocs.io/en/latest/index.html https://www.bitrook.com/
AutoClean https://github.com/elisemercury/AutoClean
Dora,PrettyPandas,DataCleaner,Tabulate,Pyjanitor
b.Handle imbalance Collect More Data if possible,Try Resampling Your Dataset
1.Under Sampling - mostly not prefer because lost of data imbalaced-learn,tomek links,Random Under-Sampling, Edited Nearest Neighbours,NearMiss
Random majority under-sampling with replacement,Tomek Links Undersampling,Under-sampling with Cluster Centroids,Condensed Nearest Neighbour,One-Sided Selection,Neighboorhood Cleaning Rule,One-Sided Selection,
2.Over Sampling (RandomOverSampler (here new points create by same dot)) , SMOTETomek(new points create by nearest point so take long time),BorderLine Smote,Borderline-SMOTE SVM,FAIR SMOTE,DBSMOTE,SMOTE-ENN ,KMeans Smote,SVM Smote,SMOTe NC,ENNSMOTE,SVMSMOTE,MOTE-N ADASYN,ADASYN,Smote-NC,Random Over Sampling,RandomUnderSampler,SMOTEN,SMOTE-Tomek,SMOTE-ENN,SMOTE-CUT,Cluster-Based Over Sampling, Informed Over Sampling,MSMOTE,Oversampling Using Gaussian Mixture Models,SMOTE + Tomek Links, SMOTE + ENN,Crucio SMOTEENN,NearMiss,OSS & NCR — under sampling,Borderline SMOTE KNN,Borderline SMOTE SVM,Adaptive Synthetic Sampling (ADASYN),BalancedBaggingClassifier() , BalancedRandomForestClassifier SMOTE-NC
Over-sampling followed by under-sampling : SMOTE + Tomek links,SMOTE + ENN
smote_variants https://github.com/analyticalmindsltd/smote_variants
https://towardsdatascience.com/5-smote-techniques-for-oversampling-your-imbalance-data-b8155bdbe2b5
https://www.analyticsvidhya.com/blog/2017/03/imbalanced-data-classification/
ensmble based -Bagging Based techniques, Boosting-Based techniques,Adaptive Boosting- Ada Boost techniques,Gradient Tree Boosting,XG Boost
tools Imb-learn,SMOTE-Variants,Regression-ReSampling https://towardsdatascience.com/tools-to-handle-class-imbalance-bff20c3bf099
Balancing data sets with Crucio ADASYN https://medium.com/softplus-publication/balancing-data-sets-with-crucio-adasyn-79f04ff0779d
LoRAS: A Better Oversampling Algorithm https://analyticsindiamag.com/hands-on-guide-to-loras-a-better-oversampling-algorithm/ https://github.com/narek-davtyan/LoRAS
https://towardsdatascience.com/7-over-sampling-techniques-to-handle-imbalanced-data-ec51c8db349f
Combining Over and Under-sampling
3.class_weight give more importance(weight) to that small class ( Cost-Sensitive Algorithms)
from sklearn import compute_class_weight
Cost-sensitive learning,Class-balanced loss,Focal loss
weighted loss function
4.use Stratified kfold to keep the ratio of classess constantly, train teat spilt startify attribute
Use K-fold Cross-Validation in the Right Way,Stratified Cross Validation,repeated K-fold Cross-Validation,Stratified K-fold Cross-Validation
Stratified Sampling,Stratified splits
5.Weighted Neural Network
cluster based sampling
6.MESA https://analyticsindiamag.com/guide-to-mesa-boost-ensemble-imbalanced-learning-with-meta-sampler/
7.choose Proper Evaluation Metric metric roc,f1,etc...
https://machinelearningmastery.com/framework-for-imbalanced-classification-projects/ https://www.kdnuggets.com/2020/01/5-most-useful-techniques-handle-imbalanced-datasets.html
8.Deep Imbalanced Regression https://github.com/YyzHarry/imbalanced-regression https://analyticsindiamag.com/deep-imbalanced-regression-complete-guide/
Imbalanced Dataset Sampler https://github.com/ufoym/imbalanced-dataset-sampler
9.Ensemble Techniques ensemble techinque - Bagging Based techniques,Boosting-Based techniques
BalancedBaggingClassifier,Threshold moving,Easy Ensemble classifier,Balanced Random Forest,Balanced Bagging,RUSBoost,MESA
10.Try Different Algorithms (ensemble techinque - Bagging Based techniques,Boosting-Based techniques)
model based (some models are particularly suited for imbalanced dataset)
Algorithmic Ensemble Techniques,Tree-Based Algorithms
11.Try a Different Perspective ( consider as anomaly detection or change detection)
Threshold Moving Methods,One-Class Classification,Customised Ensemble Algorithms
Probability Tuning Algorithms,Calibrating Probabilities,Tuning the Classification Threshold
12.databalancer https://github.com/pradeepdev-1995/databalancer
13.collect more data
14.treat problem as anomaly detection
15.Combined Class Methods
In this type of method, various methods are fused together to get a better result to handle imbalance data. For instance, like SMOTE can be fused with other methods like MSMOTE (Modified SMOTE), SMOTEENN (SMOTE with Edited Nearest Neighbours), SMOTE-TL, SMOTE-EL, etc. to eliminate noise in the imbalanced data sets
16.One-Class Algorithms,One-Class Support Vector Machines,Isolation Forests,Minimum Covariance Determinant,Local Outlier Factor,Mahalanobis Distance for One Class Classification
17.BalancedBatchGenerator https://imbalanced-learn.org/stable/references/generated/imblearn.keras.BalancedBatchGenerator.html
18.train_test_split stratify attribute , stratify split
19. https://github.com/pradeepdev-1995/databalancer
Meta’s balance package https://github.com/facebookresearch/balance
c.Remove noise data
d.Format data
d.Discretize a.Equal width binning b.Equal frequency binning c.K-means Binning d.Discretization by Decision Trees e.ChiMerge f.Arbitrary Discretization g.Quantile h.Custom Discretization
Discretisation plus categorical encoding,Discretisation plus encoding Discretisation with classification trees,Domain knowledge discretisation
Data Binning
Binning based on distribution (quantile-cut),Binning based on values (cut)
Bucketing , quantile bucketing ,Clipping
e.Handle categorical data Ordinal,Nominal,cyclic,binary categorical variables
1.One Hot Encoding , dummy, and effect coding,Similarity Encoding,Binary Encoding
Rainbow Method for Label Encoding
2.Count Or Frequency Encoding
3.Ordinal encoding,Nominal Encoding,Monotonic ordinal encoding,Target Guided Ordinal Encoding,Target Guided Mean Encoding,Target-Mean-Encoding
4.Target encoding / Mean encoding,GapEncoder,MinHashEncoder,Target guided ordinal encoding,Bayesian Target Encoding
Target Encoding,K-Fold Target Encoding,Leave-One-Out Target Encoding,Leave One fold out Target Encoding,Target Encoding with a Weighted Mean
5.Probability Ratio Encoding,Rank Encoding,Polynomial Encoding,Backward Difference Encoding
6.label encoding or .cat.codes ,Label Encoding with Rainbow Method
7.probability ratio encoding
8.woe(Weight_of_evidence)
Word2Vec(word Word embedding)
9.one hot encoding with multi category (keep most frequently repeated only) (One hot encoding of top categories)
10.feature hashing,CatBoost Encoding
11.sparse csr matrix
12.entity embeddings,Categorical Embeddings
13.binary encoding,Base-N Encoding
14.Rare label encoding
15.Leave-one-out(Loo) encoding,Generalized Linearn Mixed Model
16.hash encoding,MinHashEncoder,SimilarityEncoder,DatetimeEncoder,SuperVectorizer,FeatureHasher,DictVectorizer,HashingVectorizer,DecisionTreeEncoder
17.dummy encoding,NaN Encoding,bin counting scheme,effect coding scheme
18.Helmert Encoding,Backward Difference Encoding,James-Stein Encoding,M-estimator Encoding,Thermometer Encoder,Bayesian Encoders,Effect Encoding
Helmert Encoding,Base N Encoding,Hash Encoding,Effect or Sum or Deviation Encoding,Backward Difference Encoding,M-Estimator Encoding,James- Stein Encoding,Thermometer Encoding,CatBoost Encoding,Backward Difference Encoding,Binary Encoding,NaN encoding Polynomial encoding,Expansion encoding,Probability Ratio,Binary encoding,cat boost encoder,glm encoder,m-estimte,sum coding, polynomial Encoding,PRatioEncoder,DecisionTreeEncoder,Similarity Encoding,BackwardDifferenceEncoder GapEncoder,MinHashEncoder,TargetEncoder,Polynomial Encoding,James-Stein Encoding,MultiLabelBinarizer,SumEncoder,Quantile Encoder,Summary Encoder ,Base N Coding,Leaf Encoding,GLMM Encoding,James-Stein Encoding,Thermometer Encoding,Quantile Encoding,Summary Encoding,Collapsing Categories
Transform your categorical columns with imperio SmoothingTransformer
entity encoder for categorical variable https://contrib.scikit-learn.org/category_encoders/
Automatically selects the best encoder https://github.com/dirty-cat/dirty_cat
Improve ML Model Performance by Combining Categorical Features https://towardsdatascience.com/improve-ml-model-performance-by-combining-categorical-features-a23efbb6a215
https://towardsdatascience.com/beyond-one-hot-17-ways-of-transforming-categorical-features-into-numeric-features-57f54f199ea4
https://towardsdatascience.com/how-to-encode-categorical-data-d44dde313131 https://towardsdatascience.com/python-for-finance-7-useful-libraries-that-you-should-know-e422b9e9aaba
f.Scaling of data
1.Normalisation
2.Standardization(StandardScaler)
3.Robust Scaler not influenced by outliers because using of median,IQR
4.Min Max Scaling
5.Mean normalization
6.maximum absolute scaling
7.Power Transformer Scaler
8.Scaling To Median And Quantiles,Scaling to minimum and maximum values,Scaling to the vector norm
9.unit vector scaler
10.Z-score standardization
https://www.analyticsvidhya.com/blog/2020/07/types-of-feature-transformation-and-scaling/?utm_source=linkedin&utm_medium=KJ|link|high-performance-blog|blogs|44204|0.375
Probability and Statistics Packages : PyMC3, tensorflow-probability,Pyro,GPyTorch,hmmlearn,pomegranate,GPflow,patsy,pingouin,Orbit
Q-Q plot or Shapiro-Wilk Normality Test or lilliefors test or Jarque-Bera test or Kolmogorov-Smirnov or Anderson-Darling test is used to check whether feature is guassian or normal distributed required for linear regression,logistic regression to Improve
performance if not distributed then use below methods to bring it guassian distribution
normal test,Histogram,Q-Q plot,KDE plot,Skewness and Kurtosis for check normal distribution
Fitter Library Finding the Best Distribution that Fits Your Data https://towardsdatascience.com/finding-the-best-distribution-that-fits-your-data-using-pythons-fitter-library-319a5a0972e9
anderson teset use for check any distribution
Basic Distributions - PDF, PMF, CDF, PPF,Unform, Gaussian, Bernoulli, Multinomial,Normal Distribution,Poisson, Exponential, Geometric, Log-normal distribution, Pareto/Power Law Distribution
b.Logarithmic Transformation,LogCpTransformer
c.Reciprocal Trnasformation
d.Square Root Transformation
e.Exponential Transdormation
f.BoxCOx and Yeo-Johnson Transformation
g.log(1+x) Transformation
h.johnson
i.power transformations https://towardsdatascience.com/when-and-how-to-use-power-transform-in-machine-learning-2c6ad75fb72e
g.Quantile Transformation ,Arcsin Transformation , Inverse of Log,Inverse of Exponential,Inverse of Square Root,Square of Log,Square root of Exponential
Root transformation,Cube root transformation,Cosine Transformation,SplineTransformer,FunctionTransformer,ArcsinTransformer
Left skewness (use powers) Squares transformation,Cubes transformation,High powers
g.Remove low variance feature by using VarianceThreshold
remove Duplicate data,Low variation data,Irrelevant data,Incorrect data
remove Low entropy of categorical attributes
h.Same variable(only 1 variable) in feature then remove feature
i.Outilers removing outilers depond on problem we are solving https://github.com/jainyk/package-outlier
2 type of outilers available: Global outiler(single value/data point that deviates from the distribution), Local outiler,Contextual (conditional) outliers,Collective outliers(Group of datapoint deviates from the distribution)
eg: incase of fraud detection outilers are very important
methods to find outiler: Tukey’s fences ,KNN distance,Autoencoders,Standard Deviation,zscore,boxplot,scatter plot,histogram,Violin Plot,IQR,TensorFlow_Data_Validation,svm,One-Class SVM,Isolation Forest,kmeans,DBSCAN,K Means Clustering,Percentile,knn,autoencoder,local outiler factor,One-Class Classification,Medıan Absolute Devıatıon
Automatic Outlier Detection:Isolation Forest,DBSCAN,Local Outlier Factor,Standard Deviation Approach,K Means Clustering,Minimum Covariance Determinant,Robust Random Cut Forest,DBScan Clustering,One-Class Classification,One-Class SVM,Autoencoder,Outlier Detection using In-degree Number,Histogram-based Outlier Detection,Robust Covariance,PyNomaly,angle-based outlier detection (ABOD),k-Nearest Neighbors Detector,Elliptic Envelope,Cluster-based,Local Outlier Factor,Histogram-based Outlier Detection
outiler treatment: Keep them,mean/median/random imputation,drop,discretization (binning),Winsorization,treat as seperate group,replace with resperctive percentiles,standardize and scale the data,transformation(log,scaling,sqrt,power),Replace the outlier values with a suitable value (Like 3rd deviation),Percentile Based Flooring and Capping,Binning,Trimming,Treating outliers as missing values,Top/bottom/zero coding,winsorizing,robust scaler,log transformation,binning,regularisation,Discretization,arbitrary value
Outlier capping with IQR Outlier capping with mean and std Outlier capping with quantiles Arbitrary capping
Separation: If the amount of the outlier is higher than the normal then we can separate them from the main data and fit the model on them separately
Use a Different algorithm that is not sensitive to outliers
Segment data so outliers are in a separate group
Weighted means (which put more weight on the “normal” part of the distribution)
Trimming: Remove outliers from dataset. However, it can remove large proportion of data.
Capping: No data is removed. However, it distorts variable distribution.
Missing data: The outliers are treated as missing data.
Discretization: The outliers are put into lower and upper bins.
Arbitrary capping: Domain knowledge of the variable is required to cap the min and max
Winsorization: Truncate or cap extreme values to reduce the impact of outliers
Transformation: Apply logarithmic or square root transformations
Modeling techniques: Use robust regression or tree-based models
Outlier removal: Remove the values with careful consideration if they pose an extreme challenge
Separate Analysis : This involves performing separate analyses for the data with and without outliers
Flagging : Create an additional variable to indicate outliers, providing transparency about their presence in the dataset.
ML model which are not sensitive to outliers Like:-KNN,Decision Tree,SVM,NaïveBayes,Ensemble
PyOD: A Python Toolkit For Outlier Detection https://analyticsindiamag.com/guide-to-pyod-a-python-toolkit-for-outlier-detection/
TODS: An Automated Time-series Outlier Detection System https://github.com/datamllab/tods https://towardsdatascience.com/tods-detecting-outliers-from-time-series-data-2d4bd2e91381
anomalib anomaly detection library https://github.com/openvinotoolkit/anomalib
if outiler present then use robust scaling
alibi-detect https://github.com/SeldonIO/alibi-detect#adversarial-detection https://docs.seldon.io/projects/alibi-detect/en/latest/
https://medium.com/towards-artificial-intelligence/outlier-detection-and-treatment-a-beginners-guide-c44af0699754
https://towardsdatascience.com/two-outlier-detection-techniques-you-should-know-in-2021-1454bef89331
j.Anomaly anomaly-detection-resources https://github.com/yzhao062/anomaly-detection-resources
Types of Anomalies : Point anomalies,Contextual anomalies,Collective anomalies,Group Anomalies,Spatial Anomalies,Temporal Anomalies
clustering techniques to find it
Timetk https://towardsdatascience.com/timetk-the-r-library-for-time-series-analysis-9822f7720318
Isolation Forest(for Big Data),Z score,dbscan,Local Outlier Factor,One-Class Support Vector Machine,Autoencoders,knn,Time Series Analysis,Elliptic EnvelopeInterquartile Range,Median Absolute Deviation,K-Nearest Neighbours,Fast-MCD,Auto Encoders,K-means,Histogram-based,pca,K-means,Gaussian Mixture Model,Autoencoder,Hidden Markov Models (HMM)
𝐏𝐲𝐎𝐃
Local Correlation Integral (LCI),Histogram-based Outlier Detection (HBOS),Angle-based Outlier Detection (ABOD),Clustering-Based Local Outlier Factor (CBLOF),Minimum Covariance Determinant (MCD),Stochastic Outlier Selection (SOS),Spectral Clustering for Anomaly Detection (SpectralResidual),Feature Bagging,Average KNN,Connectivity-based Outlier Factor (COF),Variational Autoencoder (VAE)
bootstrapping to remove the influence of the outlier data
Anomaly detection using PyOD https://pyod.readthedocs.io/en/latest/ https://www.youtube.com/watch?v=QPjG_313GOw https://github.com/yzhao062/pyod https://pyod.readthedocs.io/en/latest/pyod.models.html
ADBench https://github.com/Minqi824/ADBench
Anomaly Detection Pyfbad https://github.com/Teknasyon-Teknoloji/pyfbad
divided into three types:Point/Global Anomalies,Collective Anomalies,Contextual Anomalies https://towardsdatascience.com/a-comprehensive-beginners-guide-to-the-diverse-field-of-anomaly-detection-8c818d153995
https://github.com/zhuyiche/awesome-anomaly-detection
https://medium.com/@ODSC/data-sciences-role-in-anomaly-detection-bd976f93d7e3
k.Sampling techniques
Random Sampling,Systematic Sampling,Cluster Sampling,Weighted Sampling,Stratified Sampling
a.biased sampling
b.unbiased sampling
l.Feature Creation
a.Combination of multiple features with mathematical operations
b.Combination of multiple features with a reference value
3.Exploratory Data Analysis(eda)
Explore the dataset by using python or microsoft Excel,Atoti,Power BI,Datapane’s,Tableau,TabPy,SAS Business Intelligence and Analytics Tool,QlikView,PyToQlik ,KNIME,Splunk,RapidMiner,Zoho Analytics,Sisense etc...
TabPy: Combining Python and Tableau https://www.kdnuggets.com/2020/11/tabpy-combining-python-tableau.html
atoti https://www.atoti.io/ https://www.youtube.com/watch?v=Hb6mSXa14oo Datapane’s Create a Beautiful Dashboard in Python in a Few Lines of Code https://towardsdatascience.com/datapanes-new-features-create-a-beautiful-dashboard-in-python-in-a-few-lines-of-code-a3c44523292b
Switching from Spreadsheets to Neptune.ai https://neptune.ai/blog/switching-from-spreadsheets-to-neptune-ai
Data Analysis using excel https://www.excel-easy.com/data-analysis.html https://www.educba.com/data-analysis-tool-in-excel/ https://www.youtube.com/watch?v=OOWAk2aLEfk
Power BI In Jupyter Notebooks https://github.com/microsoft/powerbi-jupyter https://analyticsindiamag.com/microsoft-releases-power-bi-in-jupyter-notebooks/
Mito Generating Python By Editing Spreadsheet https://www.youtube.com/watch?v=yy3-C39ra6s https://trymito.io/?source=twitter1
Automate Pivot Table with Python https://towardsdatascience.com/automate-excel-with-python-pivot-table-899eab993966
OpenPyXL: A Python Module For Excel https://analyticsindiamag.com/guide-to-openpyxl-a-python-module-for-excel/
causal interactive dashboards and beautiful visuals https://www.causal.app/,
Visual Programming (Orange) https://orange.biolab.si/
Integrating Tableau With Python https://analyticsindiamag.com/tabpy/ Qlib https://analyticsindiamag.com/qlib/
Data visualization (Matplotlib,Seaborn,DASH,Plotly,Plotly-Express,pyqtgraph,Bokeh,Pandas-Bokeh,Pygal,hvplot,holoviews,chartify,lets-plot,pyqtgraph,glue,plotnine,pygal,bqplot,toyplot,chart,itkwidgets,vedo,ipyvolume,pyvista,glumpy,geopandas,pycountry,geopy,geo-py,pypopulation, geotext,folium,cartopy,gmplo,ipyleaflet,geoviews,geoplot,splot,arviz, hypertools,geoplotlib,Geopandas package,choroplethmaps,Leafmap,Dash,Pydot,Geoplotlib,ggplot,visualizer,Greppo,Altair,folium,geoplot,networkx,graphviz,pydot,pygraphviz,python-igraph,pyvis,pygsp,ipycytoscape,nxviz ipydagred3,Diffbot,etc...)
Dashboarding : bokeh,dash,streamlit,panel,visdom ,voila,wave,jupyter-flex,ipyflex,pandas_bokeh
Openpxl: Automate Excel Reporting Datapane: A Python Library to Build Interactive Reports
Scatterplot,Binned Scatterplot,multi line plot,bubble chart,line charts,bar chart,histogram,boxplot, Pie charts,Line Plot,distplot,Histogram
Gantt Chart,bubble charts,area plot,heat map,index plot,violin plot,time series plot,density plot,dot plot,strip plot,plotly,Choropleth Map,Kepler,PDF,Kernel density function,networkx,Scatter_matrix,Bootstrap_plot,functionvis,Higher-Dimensional Plots,3-D Plots,3D Plots With Matplotlib,3D Plots With Plotly,Animated Plot With Plotly,Word Clouds,HoloViz,Horizontal Bar Graphs,Stacked Bar Graphs,Group Bar Graphs,Raincloud Plotsradviz,bootstrap_plot,lag_plot,JoyPy plots,Gantt Chart,Box and Whisker Plot,Waterfall Chart,Pictogram Chart,Timeline,highlight Table,Bullet Graph,Choropleth Map,Word Cloud,Network Diagram,Correlation Matrices,Bubble clouds,Cartograms,Circle views,Dendrograms,Dot distribution maps,Open-high-low-close charts,Polar areas,Radial trees,Ring Charts,Sankey diagram,Span charts,Streamgraphs,Treemaps,Wedge stack graphs, table charts,lollipop charts,distplot,floWeaver
hvplot A high-level plotting API for the PyData ecosystem built on HoloViews https://hvplot.holoviz.org/
50-charts https://towardsdatascience.com/how-did-i-classify-50-chart-types-by-purpose-a6b0aa5b812d
all in one https://app.learney.me/
Python Tool For Visualizing and Plotting 2D/3D Scientific Data https://analyticsindiamag.com/guide-to-mayavi-a-python-tool-for-visualizing-and-plotting-2d-3d-scientific-data/
patchworklib - combine multiple py charts easily
7 Techniques to Visualize Geospatial Data https://www.kdnuggets.com/2017/10/7-techniques-visualize-geospatial-data.html
data to viz https://www.data-to-viz.com/
Interactive plots directly with pandas https://towardsdatascience.com/get-interactive-plots-directly-with-pandas-13a311ebf426
Top 10 Data Visualization Tools https://www.analyticsvidhya.com/blog/2021/04/top-10-data-visualization-tools/ https://www.xenonstack.com/blog/data-visualization-tools/
https://www.analyticsvidhya.com/blog/2021/03/when-to-use-what-plot-a-beginners-guide-to-select-plots-for-visualization/
https://towardsdatascience.com/8-free-tools-to-make-interactive-data-visualizations-in-2021-no-coding-required-2b2c6c564b5b
https://datavizproject.com/ https://datavizcatalogue.com/
https://attachments.convertkitcdnm.com/232198/ee18f415-1406-4e5c-94f1-49a2c6e3ec4e/Statistics-The-Big-Picture-Poster.pdf
https://towardsdatascience.com/8-free-tools-to-make-interactive-data-visualizations-in-2021-no-coding-required-2b2c6c564b5b
HiPlot (high dimensional data)-https://github.com/facebookresearch/hiplot https://levelup.gitconnected.com/learn-hiplot-in-6-mins-facebooks-python-library-for-machine-learning-visualizations-330129d558ac
https://towardsdatascience.com/top-6-python-libraries-for-visualization-which-one-to-use-fe43381cd658
https://www.kaggle.com/abhishekvaid19968/data-visualization-using-matplotlib-seaborn-plotly
𝗞𝗲𝗿𝗮𝘀 𝗠𝗼𝗱𝗲𝗹 𝘃𝗶𝘀𝘂𝗮𝗹𝗶𝘇𝗮𝘁𝗶𝗼𝗻 𝗴𝗲𝗻𝗲𝗿𝗮𝘁𝗼𝗿(ann-visualizer)- 𝗽𝗶𝗽𝟯 𝗶𝗻𝘀𝘁𝗮𝗹𝗹 𝗴𝗿𝗮𝗽𝗵𝘃𝗶𝘇
univariate and bivariate and multivariate analysis
model visualization Tensorboard,netron,playground tensorflow,plotly,TensorDash,Dash,Microscope,Lucid
distributions(discerte,continous)
data distributions-normal distribution,Standard Normal Distribution,Student's t-Distribution,Bernoulli Distribution,Binomial Distribution,Poisson Distribution,Uniform Distribution,F Distribution,Covariance and Correlation
Pingouin statistical package https://pingouin-stats.org/index.html https://www.youtube.com/watch?v=zqi51Wu5qC0
Types of Statistics
1.Descriptive
Descriptive statistics :Mean, mode, standard deviation, median ,absolute deviation, kurtosis, skewness
2.Inferential
Types of data
1) Categorical (nomial,ordinal)
2) Numerical (discerte,continous)
random variable(discerte random variable ,continous random variable)
Quantile statistics Q1, Q2, Q3, min, max, range, interquartile range
Central Limit Theorem,Bayes Theorem,Confidence Interval,Hypothesis Testing,z test, t test,f test,Confidence Interval,1 tail test, 2 tail test,chisquare test,anova test,A/B testing
Categorical vs Categorical Chi-square test,Information gain,Cramer’s V
Categorical vs Numerical Student T-test,ANOVA,Logistic regression,Discretize Y (left column),Point-biserial correlation
Numerical vs Categorical Student T-test,ANOVA,Logistic regression,Discretize X (row above)
Numerical vs Numerical Correlation,Linear Regression,Discretize Y (left column),Discretize X (row above)
4.Feature selection https://github.com/solegalli/feature-selection-for-machine-learning
upgini Free automated data enrichment library for machine learning https://github.com/upgini/upgini https://upgini.com/
FeatureSelector https://github.com/WillKoehrsen/feature-selector feature_engine https://github.com/solegalli/feature_engine
1.Filter methods (Removing Constant feature,Removing Quasi constant feature,Removing Duplication feature,Removing Correlated Features,feature importance,chisquare test,Ttest,ftest,vif,anova test,information gain,F-score,Mutual Information,hypothesis test,information gain,Univariate Selection Methods,SelectKBest,SelectPercentile,Variance threshold,Fisher’s Score,Dispersion ratio Mean Absolute Difference (MAD), constant features elimination, quasi-constant features elimination, duplicate feature elimination,univariate method,mutual information, correlation etc...),Correlation Coefficient,Variance Threshold ,Mean Absolute Difference (MAD),Dispersion ratio,Variance inflation,factor Condition Index
2.Wrapper methods (recursive feature eliminiation,Recursive feature addition,SelectKbest,boruta,mRMR,forward feature selection,backward feature elimination,Bi-directional selection,exhaustic feature selection,stepwise selection,step forward selection,step backward selection and exhaustive search etc...)
3.Embedded method (lasso regression,ridge regression,elastic net regression,tree based(Tree-based methods like Random Forest Importance etc...),Feature Selection by Tree importance,Feature selection with decision trees,regression coefficients(logistic,linear coeffiicients),Recursive feature elimination based on importance,Least absolute deviation)
4.Hybrid Method(Recursive Feature Selection,Recursive Feature addition,Recursive feature elimination,Feature Shuffling,Feature performance,Target mean performance,Permutation importance,Population stability index,Target encoding)
unsupervised Feature selection:Principal Component Analysis,Independent Component Analysis,Non-Negative Matrix Factorization,t-distributed Stochastic Neighbor Embedding,Autoencoder
Single-Agent Reinforcement Learning Feature Selection (SARLFS) ,Multi-Agent Reinforcement Learning Feature Selection (MARLFS)
ITMO_FS is a feature selection library https://github.com/ctlab/ITMO_FS
Sparse Features - Removing features,LASSO regularization,features dense(pca,Feature hashing),Using models that are robust to sparse features
5.Feature creation
feature selection https://medium.com/analytics-vidhya/feature-selection-extended-overview-b58f1d524c1c
mrmr_selection automatic feature selection at scale https://github.com/smazzanti/mrmr
Feature selector https://github.com/WillKoehrsen/feature-selector
Simulated Annealing https://github.com/kennethleungty/Simulated-Annealing-Feature-Selection
boruta https://github.com/scikit-learn-contrib/boruta_py https://github.com/Ekeany/Boruta-Shap
DropConstantFeatures DropDuplicateFeatures DropCorrelatedFeatures
step forward feature selection https://www.kdnuggets.com/2018/06/step-forward-feature-selection-python.html
automatic feature selection mrmr https://github.com/smazzanti/mrmr
Creating New Features Deep Feature Synthesis https://docs.featuretools.com/en/v0.16.0/automated_feature_engineering/afe.html
SequentialFeatureSelector: The popular forward and backward feature selection
Alternative feature selection methods Feature shuffling,Feature performance,Target mean performance
Automatic Feature Selection : recursive feature elimination and cross-validation
Powershap: A Shapley feature selection method https://github.com/predict-idlab/powershap
VarianceThreshold,Chi-squared stats,ANOVA using f_classif,Univariate Linear Regression Tests using f_regression,F-score vs Mutual Information,Mutual Information for discrete value,Mutual Information for continues value,SelectKBest,SelectPercentile,SelectFromModel,Recursive Feature Elimination,Extra Trees model
4.Feature Importance
a.ExtraTreesClassifier,ExtraTreesregressor
b.SelectKBest
c.Logistic Regression
d.Random_forest_importance,Permutation Feature Importance
e.decision tree
f.Linear Regression
g.xgboost
h.Pearson correlation
Forward selection,Chi-square,Logit (Logistic Regression model)
5.curse of dimensionality (as dimension increases performance decreases)
6.highly correleated features then can take any 1 feature (multicollinearity)
7.dimension reduction
8.lasso regression to penalise unimportant features
9.VarianceThreshold ,selectkbest
10.model based selection
11.Mutual Information Feature Selection
12.remove features with very low variance (quasi constant feature dropping)
13.Univariate feature selection
14.importance of feature (random forest importance)
15.feature importance with decision trees
16.PyImpetus
17.drop constant features (variance=0) , Drop Highly Correlated Features
18.variance inflation factor(vif)
19.Recursive Feature Elimination RecursiveFeatureAddition
20.exchaustive feature selection
21.Statistical Methods , Hypothesis Testing ,Recursive Feature Elimination
22.Boruta https://github.com/scikit-learn-contrib/boruta_py https://analyticsindiamag.com/hands-on-guide-to-automated-feature-selection-using-boruta/
23.Sequence Feature Selection, SelectFromModel
Missing Value Ratio Analysis,Low Variance Filter,High Correlation Filter,Backward Feature Elimination,Forward Feature Elimination ,SequentialFeatureSelector
PyImpetus https://github.com/atif-hassan/PyImpetus
https://www.analyticsvidhya.com/blog/2016/12/introduction-to-feature-selection-methods-with-an-example-or-how-to-select-the-right-variables/
Automate your Feature Selection Workflow in one line of Python code https://github.com/AutoViML/featurewiz https://towardsdatascience.com/automate-your-feature-selection-workflow-in-one-line-of-python-code-3d4f23b7e2c4
https://machinelearningmastery.com/feature-selection-with-real-and-categorical-data/ https://machinelearningmastery.com/statistical-hypothesis-tests-in-python-cheat-sheet/
https://www.analyticsvidhya.com/blog/2020/10/a-comprehensive-guide-to-feature-selection-using-wrapper-methods-in-python/
https://towardsdatascience.com/5-feature-selection-method-from-scikit-learn-you-should-know-ed4d116e4172
Feature Engineering Tools https://neptune.ai/blog/feature-engineering-tools?utm_source=twitter&utm_medium=tweet&utm_campaign=blog-feature-engineering-tools
https://towardsdatascience.com/practical-code-implementations-of-feature-engineering-for-machine-learning-with-python-f13b953d4bcd
PyRasgo https://github.com/rasgointelligence/PyRasgo https://docs.rasgoml.com/rasgo-docs/?_ga=2.209281745.2123722956.1645542654-525286113.1645542654
Automated Feature Engineering Using Deep Feature Synthesis (DFS) https://heartbeat.comet.ml/introduction-to-automated-feature-engineering-using-deep-feature-synthesis-dfs-3feb69a7c00b
Automatic Feature Selection in python https://verstack.readthedocs.io/en/latest/#featureselector
rulefit https://github.com/christophM/rulefit
Featurewiz: Fast way to select the best features in a data
select best features featurewiz https://github.com/AutoViML/featurewiz
Featuretools: https://github.com/alteryx/featuretools https://analyticsindiamag.com/introduction-to-featuretools-a-python-framework-for-automated-feature-engineering/
AutoFeat: https://github.com/cod3licious/autofeat
TSFresh: https://github.com/blue-yonder/tsfresh
FeatureSelector: https://github.com/WillKoehrsen/feature-selector
unsupervised feature selection technique https://github.com/atif-hassan/FRUFS
rulefit https://github.com/christophM/rulefit
5.Data splitting
Splitting ratio of data deponds on size of dataset available
Training data,Validation data,Testing data
6.Model selection
Machine learning https://scikit-learn.org/stable/index.html
Choose the Right Machine Learning Algorithm for Your Application https://towardsdatascience.com/how-to-choose-the-right-machine-learning-algorithm-for-your-application-1e36c32400b9
Time Complexity Of Machine Learning Models -https://www.thekerneltrip.com/machine/learning/computational-complexity-learning-algorithms/
interactive tools https://github.com/Machine-Learning-Tokyo/Interactive_Tools
mindsdb In-Database Machine Learning https://github.com/mindsdb/mindsdb
HTML tables into Google Sheets -https://towardsdatascience.com/import-html-tables-into-google-sheets-effortlessly-f471eae58ac9
Machine Learning Playground https://ml-playground.com/
visual introduction to machine learning http://www.r2d3.us/visual-intro-to-machine-learning-part-1/
draw a dataset from inside jupyter https://pypi.org/project/drawdata/ https://www.youtube.com/watch?v=b0rsDPQ3bjg
Visual programming language for machine learning - Kobra https://kobra.dev/
compose generate labels for supervised learning https://github.com/alteryx/compose https://analyticsindiamag.com/guide-to-prediction-engineering-with-compose/
human-learn https://towardsdatascience.com/human-learn-create-rules-by-drawing-on-the-dataset-bcbca229f00
Microscope https://microscope.openai.com/models https://www.youtube.com/watch?v=y0-ISRhL4Ks
Ptpython Autocompletion, Autosuggestion, Docstring https://github.com/prompt-toolkit/ptpython https://towardsdatascience.com/ptpython-a-better-python-repl-6e21df1eb648
3 Tools to Track and Visualize the Execution of your Python Code https://towardsdatascience.com/3-tools-to-track-and-visualize-the-execution-of-your-python-code-666a153e435e
ML Code memory Consuming https://towardsdatascience.com/how-much-memory-is-your-ml-code-consuming-98df64074c8f
PyGrid Privacy-preserving, Decentralized Data Science https://github.com/OpenMined/PyGrid/
Best and Worst Cases of Machine-Learning Models https://medium.com/towards-artificial-intelligence/best-and-worst-cases-of-machine-learning-models-part-1-36cdb9296611
https://www.youtube.com/watch?v=mlumJPFvooQ&list=PLZoTAELRMXVM0zN0cgJrfT6TK2ypCpQdY
skater Machine Learning Model Interpretation https://towardsdatascience.com/machine-learning-model-interpretation-47b4bc29d17f
Speedml Speeding up Machine Learning https://towardsdatascience.com/speedml-speeding-up-machine-learning-5dccbf21effd
2-2000x faster ML algos https://github.com/danielhanchen/hyperlearn
snapml 30 Times Faster Than Scikit-Learn snapml https://www.zurich.ibm.com/snapml/
scikit-learn-intelex https://github.com/intel/scikit-learn-intelex
composer speed-up algorithms for model training https://github.com/mosaicml/composer
pdpipe https://github.com/pdpipe/pdpipe pipeline https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html
PHOTONAI A high level Python API for designing and optimizing machine learning pipelines https://www.photon-ai.com/
Machine Learning in Tableau with PyCaret https://towardsdatascience.com/machine-learning-in-tableau-with-pycaret-166ffac9b22e
TabNet balances explainability and model performance on tabular data https://towardsdatascience.com/tabnet-e1b979907694
FreaAI That Automatically Finds Weaknesses In ML models https://analyticsindiamag.com/ibm-launches-freaai-that-automatically-finds-weaknesses-in-ml-models/
A.Supervised learning (have label data)
Transformers for Tabular Data: TabTransformer https://github.com/lucidrains/tab-transformer-pytorch
1.Regression (output feature in continous data form)
linear regression,Multiple Linear Regression,polynomial regression,Exponential Regression,Bayesian Regression,Robust Regression,Huber regressor,support vector regression,Decision Tree Regression,Random Forest Regression,TensorFlow Decision Forests,RANSAC Regression,
least square method,linear-tree,Random Forest Regression, Regularized Greedy Forests,xgboost,ridge(L2 Regularization),lasso(L1 Regularization (more sparse)),elastic, Lars,catboost,gradientboosting,adaboost,Explainable Boosting Machine,Histogram-Based Gradient Boost,Stacked Gradient Boosting Machines,LightBoost,CatBoost, XGBoost,autoxgb,NGBoost,XBNet,Chefboost,GPBoost,Local Cascade Ensemble,Principal Component Regression,huber_regression,ransac_regression,theilsen_regression,Linear spline,Isotonic regression,Bin regression,Cubic spline,Natural cubic splin,Exponential moving average,Quantile Regression,Quantile Random Forests,Quantile GBM
elsatic net,light gbm,ordinary least squares,cart,Stepwise Regression,Multivariate Adaptive Regression Splines ,Generalised Additive Model(learn non-linear feature),tabnet,Linear Tree regression
statsassume Automating Assumption Checks for Regression Models https://github.com/kennethleungty/statsassume
Locally Weighted Linear Regression https://towardsdatascience.com/locally-weighted-linear-regression-in-python-3d324108efbf
TuringBot https://www.youtube.com/watch?v=LyKzKvjyIPo
chefboost is an alternative library for training tree-based models https://github.com/serengil/chefboost
growtrees About Cost-Aware Robust Tree Ensembles for Security Applications https://github.com/surrealyz/growtrees
2.Classification (output feature in categorical data form)
Binary,Multi-class,Multi-labe
Logistic Regression,K-Nearest Neighbors,Support Vector Machine,Kernel SVM,Naive Bayes,Decision Tree Classification,linear-tree,TensorFlow Decision Forests,
Random Forest Classification,TensorFlow Decision Forests, Regularized Greedy Forests,xgboost,DART booster,autoxgb,LightGBM,adaboost,Gradient Boost,XBNet,catboost,gaussian NB,LGBMClassifier,LinearDiscriminantAnalysis, Extreme Gradient Boosting Machine, Explainable Boosting Machine,fairgbm
,Chefboost,GPBoost,NGBoost,Local Cascade Ensemble,passive aggressive classifier algorithm,cart,c4.5,c5.0,tabnet,ExtraTreesClassifier,TabPFN
https://mlwhiz.com/blog/2019/11/12/dtsplits/?utm_campaign=the-simple-math-behind-3-decision-tree-splitting-criterions&utm_medium=social_link&utm_source=missinglettr-linkedin
4 Useful techniques avoid overfitting in decision trees https://towardsdatascience.com/4-useful-techniques-that-can-mitigate-overfitting-in-decision-trees-87380098bd3c
Machine Learning – it’s all about assumptions https://www.kdnuggets.com/2021/02/machine-learning-assumptions.html
GPBoost: A Library To Combine Tree-Boosting With Gaussian Process And Mixed-Effects Models https://analyticsindiamag.com/guide-to-gpboost-a-machine-learning-library-to-combine-tree-boosting/
Data and Concept Drift https://evidentlyai.com/blog/machine-learning-monitoring-data-and-concept-drift
B.Unsupervised learning(no label(target) data)
1.Dimensionality reduction - PCA,ppa,SVD,LDA,som,tsne,openTSNE,plsr,pcr,autoencoders,kernelpca,Latent Semantic Analysis,Factor Analysis,Locality Preserving Projections,Isometric Mapping,Multiple correspondence analysis (MCA),Multiple factor analysis (MFA),Factor analysis of mixed data (FAMD),vae,CompressionVAE,Gaussian Mixture Model,Bayesian Gaussian Mixture Model
non-linear data using Kernel PCA, Non-Negative Matrix Factorization(NMF), IsoMap, t-SNE, and UMAP,TDA(Topological Data Analysis)
t-SNE Effectively https://distill.pub/2016/misread-tsne/
2.Clustering : Centroid-based Model ,Density-based Model ,Distribution-based Model,Connectivity-based model
17 clustering https://towardsdatascience.com/17-clustering-algorithms-used-in-data-science-mining-49dbfa5bf69a
https://neptune.ai/blog/clustering-algorithms?utm_source=twitter&utm_medium=tweet&utm_campaign=blog-clustering-algorithms
classix Fast and explainable clustering based on sorting https://github.com/nla-group/classix
https://www.mygreatlearning.com/blog/unsupervised-machine-learning/?highlight=unsupervised%20machine%20learning&utm_source=GLA&utm_medium=Blog&utm_campaign=1-16th%20May
https://scikit-learn.org/stable/modules/clustering.html https://machinelearningmastery.com/clustering-algorithms-with-python/
https://towardsdatascience.com/17-clustering-algorithms-used-in-data-science-mining-49dbfa5bf69a
RFM Segmentation in E-Commerce https://towardsdatascience.com/rfm-segmentation-in-e-commerce-e0209ce8fcf6
kmodes https://www.youtube.com/watch?v=8eATPLDJ0NQ
Agglomerative Hierarchical Clustering Using AGNES Algorithm https://analyticsindiamag.com/perform-agglomerative-hierarchical-clustering-using-agnes-algorithm/
CLARANS Clustering Algorithm https://analyticsindiamag.com/comprehensive-guide-to-clarans-clustering-algorithm/
https://pub.towardsai.net/fully-explained-birch-clustering-for-outliers-with-python-2ad6243f126b
https://www.kdnuggets.com/2020/12/algorithms-explained-k-means-k-medoids-clustering.html
https://www.kdnuggets.com/2017/03/naive-sharding-centroid-initialization-method.html
CLASSIX clustering https://github.com/nla-group/classix
K-Means 8x faster, 27x lower error than Scikit-learn in 25 lines https://www.kdnuggets.com/2021/01/k-means-faster-lower-error-scikit-learn.html#.YAHAAIpnx4A.linkedin
k-Means Clustering by up to 10x Over Scikit-Learn https://towardsdatascience.com/how-to-speed-up-your-k-means-clustering-by-up-to-10x-over-scikit-learn-5aec980ebb72
3.Association Rule Learning - support,lift,confidence,leverage,Conviction,aprior,elcat,Fp-growth,Fp-tree construction,FP-Max Algorithm,association_rules,Frequent Itemset Mining,Multi-Relation Association Rules,High-order pattern discovery,K-optimal pattern discovery,Approximate Frequent Itemset,Generalized Association Rules,Quantitative Association Rules,Interval Data Association Rules,Sequential pattern mining,Hypergeometric Networks,Constraint Based Mining,Multi-level Association Rules,Fuzzy Association Rules
Sequential Patterns
Generalized Sequential Patterns (GSP)
Prefix-Projected Sequential Pattern Mining (PrefixSpan)
Sequential Pattern Discovery using Equivalent Class (SPADE)
Frequent Pattern-Projected Sequential Pattern Mining (FreeSpan)
interpretable association rule https://analyticsindiamag.com/a-guide-to-interpretable-association-rule-mining-using-pycaret/
4.Market Segmentation
Demographic Segmentation,Geographic segmentation,Firmographic segmentation,Behavioural segmentation,
4.Recommendation system - Surprise,TensorFlow Recommendation,Recmterics
competitive-recsys https://github.com/chihming/competitive-recsys
a.collaborative Recommendation system (model based, memory based(item based,user based),hybrid) user-item interaction matrix
Classification-based collaborative filtering
Model-based collaborative filtering systems(Cluster model,linear regression,Bayesian networks ,latent factor(probabilistic latent,matrix factorization(als,SGD,SVD),neural network,lda))
b.content based Recommendation system
similarity based(user-user similarity,item-item similarity)
matrix factorization(SVD and SVD++),Popularity-based recommenders
c.utility based Recommendation system
d.knowledge based Recommendation system
e.demographic based Recommendation system
f.hybrid based Recommendation system
Popularity based Recommendation system (NON-PERSONALIZED )
g.Average Weighted Recommendation
h.using K Nearest Neighbor
i.cosine distance recommender system
item2vec
j.TensorFlow Recommenders https://www.tensorflow.org/recommenders
recommenders https://github.com/microsoft/recommenders
Neural Collaborative Filtering for Personalized Ranking
AutoRec: Rating Prediction with Autoencoders Matrix Factorization
k.suprise baseline model
Context-aware Recommender Systems,Mobile Recommender Systems,Group Recommender Systems,Multi-stakeholder Recommender Systems
l.Neural Collaborative Filtering (NCF)
l.Tf-Rec TensorFlow Recommendation https://github.com/Praful932/Tf-Rec
Nvidia Merlin
m.Deep Learning Recommendation Models https://www.kdnuggets.com/2021/04/deep-learning-recommendation-models-dlrm-deep-dive.html
Restricted Boltzmann Machines,Auto-Encoders
TOROS Buffalo https://github.com/kakao/buffalo
recommenders-https://github.com/microsoft/recommenders
LightFM https://making.lyst.com/lightfm/docs/home.html
lkpy Python recommendation toolkit https://github.com/lenskit/lkpy https://analyticsindiamag.com/how-to-build-recommender-systems-using-lenskit/
torchrec https://github.com/pytorch/torchrec
PyTorch implementations of deep reinforcement learning algorithms and environments https://github.com/p-christ/Deep-Reinforcement-Learning-Algorithms-with-PyTorch
recmetrics library of metrics for evaluating recommender systems https://github.com/statisticianinstilettos/recmetrics
Downsize Recommendation Models By 112 Times https://analyticsindiamag.com/explained-facebooks-novel-method-to-downsize-recommendation-models-by-112-times/
torchrec,Lenskit,RGRecSys,Surprise,Tensorflow Recommenders,NVIDIA-Merlin,Recmetrics,Surprise,DeepCTR,OpenRec,fastFM,LightFM
Session-based RecSys could be done with:Recency-based Weighting (exp.decay),Probabilistic Graphical Models (FPMC, FOSSIL),Convolutional NN (Caser, NextItNet),Recurrent NN (GRU4Rec),Graph NN (SRGNN, GCSAN),Attention(STAMP, NARM, FDSA, SHAN),Transformer(BERT4Rec, Transformer4Rec),Knowledge Graph(KSR, GRU4RecKG, KGCN, KGAT, RippleNet),Landscape, Rexy, Tensor Recommendation Engine, Light FM, Spotlight, Case Recommender
https://analyticsindiamag.com/top-open-source-recommender-systems-in-python-for-your-ml-project/
https://towardsdatascience.com/modern-recommender-systems-a0c727609aa8
https://machinelearningmastery.com/recommender-systems-resources/
C.Ensemble methods
1.Stacking models https://www.analyticsvidhya.com/blog/2021/03/advanced-ensemble-learning-technique-stacking-and-its-variants/?
vecstack https://github.com/vecxoz/vecstack
Cascading Ensembles,Cohorted Ensembles
2.Bagging models (Bagging (with the replacement) , Pasting ( without replacement ))
3.Boosting models
4.Blending
5.Voting (Hard Voting,Soft Voting)
VOTING ENSEMBLE
Simple : Max Voting, Averaging, Weighted Averaging,Simple Average,Rank Averaging,Bayesian Model,Majority Voting
mlens ML-Ensemble – high performance ensemble learning https://github.com/flennerhag/mlens
https://analyticsindiamag.com/do-ensemble-methods-always-work/
Shapley value of players (models) in weighted voting games https://github.com/benedekrozemberczki/shapley
D.Reinforcement learning https://neptune.ai/blog/best-reinforcement-learning-tutorials-examples-projects-and-courses
2 types a)model free b)model based
gym-https://github.com/openai/gym reinforcement learning by using PyTorch-https://github.com/SforAiDl/genrl
agent,environment,policy(On-Policy vs Off-Policy),reward function,value function,state,action,episode,actor-critic
agent apply action to environment get corresponding reward so that it learn environment
How to get started with Reinforcement Learning https://gordicaleksa.medium.com/how-to-get-started-with-reinforcement-learning-rl-4922fafeaf8c
1.Q-Learning
2.Deep Q-Learning
3.Deep Convolutional Q-Learning
Deep Deterministic Policy Gradient
4.Twin Delayed DDPG,DQN,Temporal difference
5.A3C (Actor Critic) ,A2C, Soft Actor Critic (SAC),Adversarial Motion Priors (AMP),Cross-Entropy Method (CEM),Deep Deterministic Policy Gradient (DDPG),Double Deep Q-Network (DDQN),Deep Q-Network (DQN),Proximal Policy Optimization (PPO),Q-learning (Q-learning),Soft Actor-Critic (SAC),State Action Reward State Action (SARSA),Twin-Delayed DDPG (TD3),Trust Region Policy Optimization (TRPO)
6.Advantage weighted actor critic (AWAC).
7.XCS
8.genetic algorithm,sarsa,natural policy gradient,Policy Gradient Learning
https://simoninithomas.github.io/deep-rl-course/
SARSA,REINFORCE,PPO,DDPG,Ddpg,TD3
AUTORL: AUTOML FOR RL https://www.automl.org/blog-autorl/
Environments-OpenAI Gym, DeepMind Lab, Unity ML-Agents
https://data-flair.training/news/python-libraries-for-reinforcement-learning/
https://analyticsindiamag.com/8-best-free-resources-to-learn-deep-reinforcement-learning-using-tensorflow/
https://analyticsindiamag.com/top-8-autonomous-driving-open-source-projects-one-must-try-hands-on/
https://analyticsindiamag.com/8-toolkits-for-reinforcement-learning-models-that-make-reasoning-explainability-core-to-ai/
https://neptune.ai/blog/best-reinforcement-learning-tutorials-examples-projects-and-courses
https://towardsdatascience.com/value-based-methods-in-deep-reinforcement-learning-d40ca1086e1
https://neptune.ai/blog/best-reinforcement-learning-tutorials-examples-projects-and-courses?utm_source=twitter&utm_medium=tweet&utm_campaign=blog-best-reinforcement-learning-tutorials-examples-projects-and-courses
TensorForce: A TensorFlow-based Reinforcement Learning Framework https://analyticsindiamag.com/guide-to-tensorforce-a-tensorflow-based-reinforcement-learning-framework/
Decision Transformer: Reinforcement Learning via Sequence Modeling https://github.com/kzl/decision-transformer
Open AI Gym - https://gym.openai.com/
DeepMind’s MuZero https://deepmind.com/blog/article/muzero-mastering-go-chess-shogi-and-atari-without-rules?utm_campaign=Learning%20Posts&utm_content=150411901&utm_medium=social&utm_source=twitter&hss_channel=tw-3018841323
KerasRL https://github.com/keras-rl/keras-rl
pyqlearning
tensorforce https://tensorforce.readthedocs.io/en/latest/index.html
Practical_RL https://github.com/yandexdataschool/Practical_RL
rl_coach https://github.com/IntelLabs/coach#installation MushroomRL https://mushroomrl.readthedocs.io/en/latest/
TFAgents https://github.com/tensorflow/agents (https://www.tensorflow.org/agents) https://deepmind.com/blog/article/trfl
TorchRec https://pytorch.org/blog/introducing-torchrec/ TensorFlow Recommenders https://www.tensorflow.org/recommenders
behaviour trees used in reinforcement learning https://analyticsindiamag.com/how-are-behaviour-trees-used-in-reinforcement-learning/
Automate The Stock Market Using FinRL (Deep Reinforcement Learning Library) https://analyticsindiamag.com/stock-market-prediction-using-finrl/
Stable Baselines https://github.com/openai/baselines
https://www.youtube.com/playlist?list=PL_iWQOsE6TfURIIhCrlt-wj9ByIVpbfGc
https://neptune.ai/blog/the-best-tools-for-reinforcement-learning-in-python?utm_source=twitter&utm_medium=tweet&utm_campaign=blog-the-best-tools-for-reinforcement-learning-in-python
Semi-Supervised Learning-small amount of labeled data with a large amount of unlabeled data during training
Machine Learning with Graphs http://web.stanford.edu/class/cs224w/
E.Deep-learning (use when have huge data and data is highly complex and state of art for unstructured data) https://www.kdnuggets.com/2019/11/designing-neural-networks.html
Model Zoo Discover open source deep learning code and pretrained models https://modelzoo.co/
Visualizing your Neural Network with Netron,Net2Vis,visualkeras,draw_convnet,NNSVG,PlotNeuralNet,Tensorboard,Caffe,Matlab,Keras.js,keras-sequential-ascii ,Netron,DotNet,Graphviz ,Keras Visualization,Conx,ENNUI,NNet,GraphCore ,Monial,Quiver
Sharing the best resources on various machine learning topics https://www.backprop.org/
deeplearning-models-https://github.com/rasbt/deeplearning-models
Deep-Learning-with-PyTorch- https://pytorch.org/assets/deep-learning/Deep-Learning-with-PyTorch.pdf
Frameworks:Pytorch,Tensorflow,Keras,caffe,theano,MXNet,Matlab,Microsoft Cognitive Toolkit,opacus(Train PyTorch models with Differential Privacy)
https://towardsdatascience.com/the-mostly-complete-chart-of-neural-networks-explained-3fb6f2367464 https://docs.deepstack.cc/getting-started/index.html
fastest way to build, debug, and interpret neural networks https://www.perceptilabs.com/
Nengo: A New Neural Network Building and Deployment Tool https://pub.towardsai.net/nengo-a-new-neural-network-building-and-deployment-tool-66677c65fa19
Binarized Neural Network memory size is reduced, and bitwise operations improve the power efficiency https://neptune.ai/blog/binarized-neural-network-bnn-and-its-implementation-in-ml
paddlehub https://github.com/PaddlePaddle/PaddleHub Performing Computer Vision & NLP Tasks in a Single Of Code https://towardsdatascience.com/performing-computer-vision-nlp-tasks-in-a-single-of-code-f7205f212d34
scikit-neuralnetwork https://towardsdatascience.com/the-simplest-way-to-train-a-neural-network-in-python-17613fa97958 https://github.com/aigamedev/scikit-neuralnetwork
NVIDIA’s Kaolin: A 3D Deep Learning Library https://analyticsindiamag.com/nvidias-kaolin-3d-deep-learning-library/ https://github.com/NVIDIAGameWorks/kaolin
PySyft is a Python library for secure and private Deep Learning https://github.com/OpenMined/PySyft
keras-vis Visualizing Learning of a Deep Neural Network https://towardsdatascience.com/deep-learning-model-visualization-6cf6290dc981
Deep Replay Visualizing Learning of a Deep Neural Network https://towardsdatascience.com/visualizing-learning-of-a-deep-neural-network-b05f1711651c
keras-visualizer Visualizing Keras Models https://towardsdatascience.com/visualizing-keras-models-4d0063c8805e
Lucid Library is an open source framework to improve the interpretation of deep neural networks
Gradient-Centralization-TensorFlow improve your training performance of TensorFlow models with just 2 lines of code! https://github.com/Rishit-dagli/Gradient-Centralization-TensorFlow
XBNet: An Extremely Boosted Neural Network
MIL-WebDNN Fastest DNN Execution Framework on Web Browser https://mil-tokyo.github.io/webdnn/
Vector Hub models to turn data into vectors text2vec, image2vec, video2vec, graph2vec, bert, inception, etc https://github.com/RelevanceAI/vectorhub
torchbearer: A model fitting library for PyTorch https://github.com/pytorchbearer/torchbearer
1.Multilayer perceptron(MLP)
1.Regression task
2.Classification task
Tabnet and deep tables for tabular dataset using deep learning
2.Convolutional neural network ( use for image data)
Best MLOps Tools for Your Computer Vision Project Pipeline https://neptune.ai/blog/best-mlops-tools-for-computer-vision-project?utm_source=twitter&utm_medium=tweet&utm_campaign=blog-best-mlops-tools-for-computer-vision-project
mediapipe https://google.github.io/mediapipe/ cv modelhub https://modelplace.ai/
all openmmlab https://github.com/open-mmlab mmdetection,mmsegmentation,mmediting,mmdetection3d,mmaction2,mmocr,mmpose,etc...
glasses High-quality Neural Networks for Computer Vision https://github.com/FrancescoSaverioZuppichini/glasses
IceVision https://airctic.com/0.8.0/
Top Computer Vision Google Colab Notebooks- https://www.qblocks.cloud/creators/computer-vision-google-colab-notebooks
for low code object detection (detecto)- https://github.com/alankbi/detecto
CV-pretrained-model- https://github.com/balavenkatesh3322/CV-pretrained-modelCV-pretrained-model-
Fast Computer Vision Model Building PyTorch Lightning Flash and FiftyOne https://towardsdatascience.com/open-source-tools-for-fast-computer-vision-model-building-b39755aab490
5 Open-Source Facial Recognition https://medium.com/analytics-vidhya/ways-to-boost-your-computer-vision-projects-by-using-5-open-source-facial-recognition-projects-56668f170cb9
cnn alternative CapsNet https://github.com/XifengGuo/CapsNet-Keras
EDA for image data data-gradients
1.Classification of image
albumentations https://github.com/albumentations-team/albumentations AugLy https://github.com/facebookresearch/AugLy
create own model,Lenet,Alexnet,DenseNet,MobileNet,ShuffleNet,SqueezeNet,Resenet,GoogleNet,Inception,Vgg16,vgg19,Efficient,EfficientNetV2,EfficientDet,residualnet,Nasnet,STN,nasneta,senet,amoebanetc,DeiT (tiny,small,base),Meta Pseudo Labels,res-mlp-pytorch,MLP-Mixer,vit,DynamicViT, FNet,gMLP models,nfnet
mmclassification https://github.com/open-mmlab/mmclassification
https://theaisummer.com/cnn-architectures/ https://paperswithcode.com/sota/image-classification-on-imagenet
timm https://pypi.org/project/timm/ https://github.com/rwightman/pytorch-image-models
2.Localization of object in image
3.Object detection and object segmentation
rcnn,fastrcnn,fastercnn,TensorFlow Object Detection,yolo v1,yolo v2,yolo v3,SlimYOLOv3,yolo v4,PP-YOLO,scaled yolov4,YOLOR,YoloV5,YOLOS,efficinetdet,fast yolo,yolo tiny,yolo lite,yolo tiny++,yolo act++,yolonas,yolov8
maskrcnn,DeepLab-v3-plus,ssd,detectron,detectron2,D2Go,mobilenet,retinanet,R-fcn,Libra_R-CNN,detr facebook,mdetr,pspnet,segnet,U-net,UNet++,Efficient U-Nets, 𝗗𝗲𝗻𝘀𝗲-𝗚𝗮𝘁𝗲𝗱 𝗨-𝗡𝗲𝘁, nnU-Net,v-net,TransUNet, H-DenseUNet, MultiResUNet ,deeplab,globalconvolutionnetwork,fcn,EfficientDet,Vision Transformer,deit,VarifocalNet (VF-Net),DINO,BodyPix,vit,AugFPN,mlsd
PixelLib Simplifying Object Segmentation with PixelLib Library https://github.com/ayoolaolafenwa/PixelLib
mmdetection https://github.com/open-mmlab/mmdetection https://towardsdatascience.com/mmdetection-tutorial-an-end2end-state-of-the-art-object-detection-library-59064deeada3 https://github.com/open-mmlab/mmrotate
mmdetection3d https://github.com/open-mmlab/mmdetection3d mmsegmentation https://github.com/open-mmlab/mmsegmentation
fewshot https://github.com/open-mmlab/mmfewshot
Zero-Shot Object Detection , annotate dataset https://github.com/microsoft/GLIP
imageai.Detection ObjectDetection Segmentation models https://github.com/qubvel/segmentation_models
Image-Segmentation-Using-Pixellib
IceVision https://airctic.com/0.8.0/
Image Generation Using TensorFlow Keras https://analyticsindiamag.com/getting-started-image-generation-tensorflow-keras/
Video Understanding https://towardsdatascience.com/video-understanding-made-simple-with-pytorch-video-and-lightning-flash-c7d65583c37e
Getting Started With Object Detection Using TensorFlow https://analyticsindiamag.com/object-detection-using-tensorflow/
Instance Segmentation using Mask-RCNN with PixelLib and Python https://www.youtube.com/watch?v=i_-ud01wFhc
MLP MLP solution for Vision, from Google AI https://github.com/lucidrains/mlp-mixer-pytorch
MMDetection https://analyticsindiamag.com/guide-to-mmdetection-an-object-detection-python-toolbox/ mediapipe https://github.com/google/mediapipe
SSL Framework For Object Detection https://analyticsindiamag.com/googles-stac-ssl-framework-for-object-detection/
GSDT https://analyticsindiamag.com/gsdt-gnns-for-simultaneous-detection-and-tracking/
D2Go Brings Detectron2 To Mobile https://analyticsindiamag.com/facebooks-d2go-brings-detectron2-to-mobile/
AdelaiDet open source toolbox for multiple instance-level detection and recognition tasks https://github.com/aim-uofa/AdelaiDet
3d object detection https://omdena.com/blog/3d-object-detection/?utm_source=linkedin&utm_medium=organic&utm_campaign=blog&utm_term=google-analytics
PyMAF https://analyticsindiamag.com/guide-to-pymaf-pyramidal-mesh-alignment-feedback/
3 kind of object segmentation are available semantic segmentation,instance segmentation,panoptic segmentation
segmentation_models https://github.com/qubvel/segmentation_models
https://analyticsindiamag.com/guide-to-panoptic-segmentation-a-semantic-instance-segmentation-approach/ https://analyticsindiamag.com/semantic-vs-instance-vs-panoptic-which-image-segmentation-technique-to-choose/
ResNeSt: A Better ResNet with the Same Costs https://analyticsindiamag.com/guide-to-resnest-a-better-resnet-with-the-same-costs/
PAN: Pyramid Attention Network for Semantic Segmentation https://medium.com/mlearning-ai/review-pan-pyramid-attention-network-for-semantic-segmentation-semantic-segmentation-8d94101ba24a
PyTorch based low code object detection-https://github.com/alankbi/detecto
https://www.kdnuggets.com/2021/03/extraction-objects-images-videos-5-lines-code.html
autogluon
GluonCV https://medium.com/apache-mxnet/start-fitting-cv-models-like-scikit-learn-with-gluoncv-0-10-931ff910a38
https://awesomeopensource.com/project/hoya012/deep_learning_object_detection
4.objecttracking (mean shit and optical flow and kalman filter)
Tracktor++,Trackrcnn,Jde,DeepSORT,FairMOT
mmtracking https://github.com/open-mmlab/mmtracking https://github.com/open-mmlab/mmflow
mmhuman3d https://github.com/open-mmlab/mmhuman3d
Video Understanding https://github.com/open-mmlab/mmaction2
5.Deepdream,Neural style transfer, Pose estimation
generative models https://github.com/open-mmlab/mmgeneration
Machine Learning for Art https://ml4a.net/#
Pose estimation by mediapipe library https://google.github.io/mediapipe/ https://www.youtube.com/watch?v=brwgBf6VB0I
posemodule https://www.youtube.com/watch?v=5kaX3ta398w Pose Tracking https://www.youtube.com/watch?v=0JU3kpYytuQ&t=1650s
6.DEEP LEARNING METHODS FOR 2D :OpenPose,DeepPose,AlphaPose,tfpose,MultiPoseNet,AlphaPose,Movenet lighting,VIBE,DeeperCut,Mask RCNN,DeepCut,Convolutional Pose Machines,PoseNet,MoveNet,Adobe’s BodyNet,MoveNet and TensorFlow.js,High-Resolution Net,Blaze pose,Deep Pose,PoseNet
openpose wrnchai densepose
mmpose https://github.com/open-mmlab/mmpose
Pose Estimation using OpenCV https://www.analyticsvidhya.com/blog/2021/05/pose-estimation-using-opencv/
https://medium.com/beyondminds/an-overview-of-human-pose-estimation-with-deep-learning-d49eb656739b
3D POSE ESTIMATION
3D Image Classification https://keras.io/examples/vision/3D_image_classification/
TensorFlow 2 Object Detection API tutorial https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/
https://blog.paperspace.com/how-to-train-scaled-yolov4-object-detection/
Image DA libraries – Augmentor, Albumentations, ImgAug, AutoAugment, Transforms https://neptune.ai/blog/data-augmentation-in-python
Simple transformations-Resize,Gray Scale,Normalize,Random Rotation,Center Crop,Random Crop,Gaussian Blur
Position augmentation-Scaling,Cropping,Flipping,Padding,Rotation,Translation,Affine transformation,Kernel filters
Color augmentation-Brightness,Contrast,Saturation,Hue
Deep learning approach-Adverserial training,Neural style transfer,Gan data argumentation
AS-One Run YOLOv7,v6,v5,R,X in under 20 lines of code https://github.com/augmentedstartups/AS-One
Data augmentation feature space : noise,interpolation Data Space Character Level : Noise Induction,Rule-based Tr