Awesome LEGO Machine Learning

A curated list of resources dedicated to Machine Learning applications to LEGO bricks.

Parts Classification

Applications

• Brickognize [2022.12], Talk from the author [2022.07] - Web application which recognizes any Lego part, minifigure, or set.
• Bricksee [2022.11] - Mobile application for organizing a Lego collection with a feature to detect parts from a photo.
• BrickMonkey [2022.11] - Mobile application to recognize minifig and parts.
• RebrickNet [2022.01] - Web application on rebrickable.com. Currently, it detects and recognizes 300 different parts and an unspecified number of colors. Train using real videos and photos submitted by users.
• Minifig Finder [2021.12] - Web application for minifig identification. Uses Mask R-CNN for detecting individual parts (head, torso, and legs) and metric learning for classification. Currently, it looks abandoned and not working.
• Unnamed mobile app [2021.10], Update 1 [2021.11], Update 2 [2021.12] - Unnamed and unreleased mobile app. Uses LDraw renders for training a classifier. The first version recognized only six different parts.
• Brickit [2021.07] - Mobile application for detecting parts and suggesting models which can be built from those parts.
• Brickly [2021.04] - Unfinished and unreleased app for part detection and identification.
• Instabrick, Kickstarter [2019.10], Review [2021.04] - Camera and web application for part identification and inventory.
• BrickBanker [2020.12] - Mobile application which detects up to 2k different parts.

Sorting Machines

• DIY LEGO Sorting Machine Backgrounds on design choices [2023.12] - The article looks into the details and individual components of our specific sorting machine, explaining the rationale behind various design decisions
• Exploring LEGO Sorting Machines: A Survey of Designs [2023.12] - The article provides a comprehensive overview of the wide range of existing LEGO sorting machines.
• BrickSortingMachine [2023.08], Blog, Code, LEGO build instructions - A LEGO brick sorting machine.
• Lego Sorting Machine [2023.08] - Sorting machine [in progress].
• Nexus [2023.03] - Open-source sorting machine with CAD designs and code available.
• Standard Sorter v1.0 [2023.01] - The robotic arm that can sort based on color and/or size.
• Eppos Sorting Machine [2023.01] - Industrial sorting machine built for 3Bricks.
• Big Robot [2022.06] - Initiative to build Lego AI Open Source Sorting Machine.
• Universal LEGO Sorting Machine [2022.06], Description - Sorting machine built for BSL Bricks store.
• Deep Learning Lego Sorter [2021.11] - A sorting machine built from Lego with instruction and code available.
• Lego Automatic Sorting LegoLAS 2.0 [2021.08], Description (in German), CAD - Student project in the Laboratory for Computer Science in Engineering and Computational Mathematics.
• DISBY Sorting Machine [2021.01] - DISBY is the world's first automated Lego brick sorting system. It features a novel AI system that recognizes bricks based on a minimal formal description retrieved from the internet (e.g. size, weight, and description).
• The World's First Universal LEGO Sorting Machine [2019.12], Classifier [2019.12], Dataset [2019.03], CV Pipeline [2019.08] - Sorting machine built by Daniel West.
• The Shape Sifter [2019.06], Blog - The Shape Sifter is a Lego sorting machine utilizing a neural network, image processing software, a conveyor belt, and air jets.
• Lego Sorter using TensorFlow on Raspberry Pi [2018.09] - Sorting machine that recognizes 11 different parts.
• Letzgo Sorter [2018.01] - Sorting machine built for Letzgo company. There are a few more videos available but not many details.
• Automatic Lego Sorting Machine - Students' report containing the design of the sorting machine. Mostly related to the mechanical side of the machine.
• Sorting 2 Metric Tons of Lego [2017.04], Software Side [2017.05] - One of the first publically described sorter that uses ML for part classification.

Other projects

• JetClean [2022.10] - JetClean is a small robotic Lego cleaner capable of autonomously navigating around your bedroom and keeping it tidy!

Code

• Setognize [2024.06] - Code for a mobile app that can recognize 50 dfferent Lego sets.
• LegoSorter [2023.09] - Code for a mobile app that can recognize and count Lego bricks. It includes scripts for rendering the dataset, training the model, and the backend for the app.
• OpenBlok [2022.11] - OpenBlok is an open-source Lego identification and sorting system using AI models developed by blokbot.io
• Lego Brick Recognition [2020.03] - Code for generating synthetic dataset and training a classifier for 15 different parts.
• Lego Classifier [2019.08] - Detailed description of training a part classifier and deploying it on Arduino.
• Lego Detector [2019.03] - Code for training a classifer.

Papers

• Brickinspector [2023.02], Paper - The paper describes the process of using synthetic data for training semantic segmentation model for detecting Lego parts.
• How to sort them? A network for LEGO bricks classification [2022.07] - The paper presents a comparison of 28 models used for image classification trained to recognize 447 different LEGO bricks.
• Hierarchical 2-step neural-based LEGO bricks detection and labeling [2021.04] - The paper proposes two-step system for identifying LEGO bricks -- detection and classification. The model is limited to recognizing only 10 different parts.
• Lego Recognition Tool [2018.03] - The diploma thesis describes the process of developing a machine, that is capable of identifying Lego bricks and sorting them into boxes.

Datasets

• B200 LEGO Detection Dataset [2024.03] - Dataset for parts detection. It contains 2k high-quality renders for 200 different parts.
• Photos and rendered images of LEGO bricks [2023.11] - The paper describes a collection of datasets containing both LEGO brick renders and real photos. The datasets contain around 155,000 photos and nearly 1,500,000 renders.
• Video of LEGO bricks on conveyor belt [2022.01] - The dataset contains videos of LEGO bricks moving on a white conveyor belt to train a classifier for sorting machine.
• B200C LEGO Classification Dataset [2021.08], Code - Dataset for parts classification. It contains 800k high-quality renders for 200 different parts.
• LEGO bricks for training classification network [2021.06] - The dataset part classification. It contains images of 447 different parts, both real photos (52k) and renders (567k).
• Tagged images with LEGO bricks [2021.02] - The dataset for parts detection. It contains 2933 photos and 2908 renders annotated with bounding boxes. It doesn't include information about part IDs.
• Lego Brick Sorting [2018.12] - The dataset for parts classification. It contains 4,580 photos of 20 different parts.
• Lego vs. Generic Brick [2018.12] - The dataset for recognizing original vs fake bricks. It contains 12 classes across 6 brick types, and more than 20k images taken by 4 cameras.

Rendering Parts

• BrickRenderer [2023.06] - Render realistic training images of ldraw pieces.
• Lego Rendering Pipeline [2023.06] - Rendering pipeline for semi-realistic, individual parts.
• Lego multi object detection [2021.11] - Script to generate renders of LEGO parts and corresponding bounding boxes. Uses Python and Blender.
• BrickRegistration [2021.11] - A tool to generate synthetic 3d scenes with LEGO parts and their segmentation information.
• Lego Renderer for ML Projects [2020.01] - A set of Python scripts/Blender utilities for rendering Lego scenes for use in deep learning/computer vision projects. Includes a basic scene with a tracked camera, scripts for rendering images, normals, masks of Lego combinations, and utilities for recording the positions of special features on different pieces (studs, corners, holes) easily.
• Rendering LDraw Parts Images for Rebrickable [2018.10] - Comprehensive guide describing how Rebrickable rendered their images.

Understanding Part IDs and colors

• Rebrickable numbering [2023.02]
• Understanding LEGO part numbers [2020.10]
• The curious case of LEGO colors [2016.09] - This post explores the difficulty of defining the exact LEGO color palette. It discusses the various lists, conversions, and color systems.
• Let's talk about classification of parts
• LDraw Part Number Specification
• Bricklink Item Numbers

Generating Sets

Papers

• Brick Yourself within 3 Minutes [2022.05] - Given the input portrait, it uses several deep neural networks to extract the attributes to describe the human appearance shown in the image. Built on these attributes, the model generates the corresponding uncolored brick model by iteratively searching for brick components with the coordinate descent algorithm. Finally, it assigns a color to every brick to get the final brick model.
• Brick-by-Brick: Combinatorial Construction with Deep Reinforcement Learning [2021.10] - Generating sets step by step from the 2D images using RL.
• Image2Lego: Customized LEGO® Set Generation from Images [2021.08] - The system takes an image as an input, encodes it as embedding, generates a voxelized 3D model, and finally convert it into LEGO bricks. The model jointly trains a 3D model autoencoder and a 2D-to-3D encoder.
• Automatic Generation of Vivid LEGO Architectural Sculptures [2019] - It introduces an automatic system to convert an architectural model into a LEGO sculpture while preserving the original model’s shape features like repeating components, shape details and planarity.
• Legolization: Optimizing LEGO Designs [2015] - It converts a 3D model into a voxelized 3D model, and then into LEGO bricks. The paper focuses on optimizing the physical stability of the generated design.
• Survey on Automated LEGO Assembly Construction [2014] - The survey paper on converting 3D models into LEGO models.

Datasets

• Lego sets made in LDraw - Links to over 2000 LEGO sets recreated in LDraw.
• LDraw Official Model Repository - LDraw repository of over 1800 official LEGO sets.
• BrickHub - Almost 500 original and custom sets.

Assembling Sets

Papers

• Robotic LEGO Assembly and Disassembly from Human Demonstration [2023.05] - This paper presents a system that automates LEGO assembly and disassembly by learning from human demonstration.
• Planning Assembly Sequence with Graph Transformer [2022.10] - A graph transformer-based framework for ASP problem, with a heterogeneous graph attention network to encoder the models, which are decoded with the attention mechanism to generate assembly sequence.
• Translating a Visual LEGO Manual to a Machine-Executable Plan [2022.07] - Understanding the assembly process using manual.
• Break and Make: Interactive Structural Understanding Using LEGO Bricks [2022.07] - Introduction of a new task for visual understanding, 3D simulator to manipulate LEGO models, and a model to solve the proposed task of recreating the LEGO model.
• Building LEGO Using Deep Generative Models of Graphs [2020.12] - It proposes a way to represent the LEGO model as graphs and learn how to generate them step-by-step.

Generating images of LEGO

Models

• Lego Minifig XL [2023.08] - Stable Diffusion model for generating Minifigures.
• Lego Brickheadz XL [2023.09] - Stable Diffusion model for generating Brickheadz.

Posts

• The AI Revolution: How Artificial Intelligence Is Impacting the LEGO Community [2023.11] - The post discusses how AI-generated “LEGO” sets are sparking a debate between inspiration and imitation within the AFOL community.
• Reimagining LEGO sets [2023.03] - Generating realistic versions of LEGO sets.
• LEGO Stable Diffusion [2023.01] - Fine-tuned stable diffusion model for generating images in the LEGO style.
• Generating LEGO Pirates sets and minifigures [2023.01] - Generating LEGO Pirates sets and minifigures using Stable Diffusion.
• Using AI to generate minifigures [2020.07], Part 2 [2020.08], Part 3 [2020.08] - Using different GANs to generate images of minifigures.

3D Reconstruction

Papers

• MobileBrick: Building LEGO for 3D Reconstruction on Mobile Devices [2023.03] - A dataset of 153 Lego sets focusing on detailed 3D object reconstruction with a unique data modality of high-resolution RGB images with low-resolution depth maps captured on a mobile device.

Price Prediction

Code

• Lego Learning: Using Machine Learning to Predict the Future Value of Lego Sets [2023.11] - Using machine learning algorithms to predict future prices of sets.

ML at LEGO Group

• Weeding out or picking winners in open innovation? Factors driving multi-stage crowd selection on LEGO ideas [2023.12] - Very detailed analysis of the factors behind the winning entries on the LEGO Ideas platform.
• Building a GenAI Solution — A Hackathon Success Story! [2023.11] - Description of the winning project of the internal Generative AI Hackathon at the LEGO Group. It focuses on generating a set description based on metadata and an image.
• AI Summit London: Lego’s Brian Schwab on Interaction Design [2023.06] - A short talk with Brian Schwab without many details about the augmented reality for playing with Lego bricks.
• A One-Stop Data Shop: The Lego Group’s Anders Butzbach Christensen [2023.03] - Podcast (and transcript) about LEGO data platform.
• How LEGO plays with data: An interview with chief data officer Orlando Machado [2022.12]
• Inside the annual LEGO Brick Hack [2022.11] - Short description of internal Data Science hackathon.
• Building Blocks of Machine Learning at LEGO with Francesc Joan Riera [2021.11] - Podcast about using ML at LEGO. In particular, their ML infrastructure, content moderation system, and classifying images of LEGO models into thematical categories.

Other lists

• Awesome LEGO - General list of LEGO resources.