Card-Playing Robot
Many people enjoy playing cards, but sometimes there isn't enough of a player!

Card-Playing Robot
Project Description
Card games are a popular pastime for people of all ages. Depending on the card game, a specific number or a minimum number of players is required; otherwise, the game cannot be played. Since the required or desired number of players is not always available, the goal of Systems Engineering 1 is to develop a robotic system that can replace a human player for playing various card games with real playing cards.
People Involved
Student Group:
Central Game Control: Markus Wagner, Marco Westermair, Dominik Ziegler
Human-Machine Interaction: Tobias Ertle, Manuel Mikschl, Alexander Markidonow
Card Recognition: Benedikt Degenstein, Ali Gözen, Thomas Henning, Dennis Religa, Burak Saglam
Robot Control: Michael Durach, Thomas Höger, Jan Sigg, Nikolas Voss
Advisors: Prof. Dr.-Ing. Martin Bayer, Prof. Dr.-Ing. Claudia Meitinger
Course
Systems Engineering 1 (E5-IK)
Duration
Winter Semester 2019/2020
Objective
Development of a functional prototype of a card-playing robot,
- which can replace a human player in the Bavarian card game "Watten" for two players.
- that can play with real playing cards.
- that can be easily expanded or modified (e.g., to support other card games, other technologies for card recognition and/or card handling, etc.).
System Overview
Functionalities to Be Implemented
To achieve the objective, the following technical functionalities should be implemented:
- Central game control system to manage the other game components
- Interaction with human players whenever appropriate or necessary
- Selecting and playing a card when it is the robot player’s turn (at a minimum, in accordance with the rules; if possible, using a simple tactic)
- Interaction with the human player(s) via various modalities
- Identifying playing cards
- Face-down cards—i.e., the robot player’s own cards that have not yet been played and should not be visible to human players
- face-up cards, i.e., the most recently played card (regardless of which player played it)
- Picking up and moving the robot
player’s playing cards- from the deck at the start of the game
- during the game, from a card holder
Arrangement of System Components
The following figure shows the selected player arrangement as well as the defined storage locations for playing cards before and during the game. By OpenPNP itself.
Modularization
In accordance with the technical functionalities to be implemented, the project was carried out as four subprojects. Each functionality was implemented on a Raspberry Pi as part of an MQTT network.
Procedure
The project was implemented in several phases over the course of the semester.
Setting up the project
- Learning about the Wadden Sea
- Creating the requirements and specifications
- Planning the structure, work packages, workflow, project plan, and costs
- Defining synchronization milestones
- Initialize Git repositories
Perform a full-system simulation
Objectives:
- Define a binding interface d (here: MQTT topics)
- Reach a common understanding of the overall system behavior
Procedure:
- Each group simulates the behavior of its subsystem.
- Communication between subsystems/groups is permitted exclusively via MQTT (e.g., using command-line tools).
Integrate the system
- During each system integration, an attempt is made to run through a complete game sequence.
- Errors and opportunities for optimization are documented and subsequently addressed by the groups.
- At the next weekly meeting, the improvements and enhancements are tested again in the form of a complete game run.
Conclude the project
- Presentation of the results of the subprojects
- Documentation of the modules
- Acceptance / Handover
Subproject: CENTRAL GAME CONTROL
Tasks
- Central control of gameplay
- Interaction with human players whenever appropriate or necessary
- Requesting robot actions
- Requesting map recognition
- Implementation of game logic
- Determining which card to play: in accordance with the rules and using a simple game strategy
Module Description
A Python script was developed to control communication among the groups, manage the game flow, and implement the strategy. A systemd service automatically starts the scripts at system startup, eliminating the need for manual intervention.
To ensure flexibility for future changes, an object-oriented design was implemented in which player types (human/robot) can be swapped just as easily as the type of game (Watten/...) or the method of communication between components.
The card-selection function assigns a value to each card, which is taken into account during decision-making. In addition, a simple heuristic is applied when choosing the play and suit, enabling the system to win games.
In addition, an MQTT logger and a simulation program were created for internal testing. The messages sent by all system components were logged to facilitate troubleshooting. The simulation program allows for testing of the team’s own software without relying on other groups.

Subproject: HUMAN-MACHINE INTERACTION
Responsibilities
- Design of the interaction concept in consultation with the central game control system
- Implementation of voice input, voice output, and touch interaction
Requirements (selection)
- The interaction should be self-explanatory for human players, i.e., usable without instruction.
- The system’s state—in terms of game progress and selected parameters (e.g., shot type, ball color)—should be visible to the player.
- The robot should react to game-critical actions in a human-like manner.
Implementation
- Programming language: Python 3
- Touchscreen display as the
basic building block, with voice output and voice input as supplementary features - Paho-MQTT library for MQTT messages
- Startup script to start/restart all programs as individual tasks (voice input, voice output, and display)
- Kivy library for the user interface
- Text-to-speech using Eclipse
- Voice input using Snips
Usage
After the Raspberry Pi 3 boots up, the Python startup script must be executed. This script then waits for commands arriving via MQTT.
As soon as user input is required, the player is prompted via voice output and on-screen instructions.
In addition to touch input, this is also possible via voice input.
Subproject: MAP RECOGNITION
Task
- Identification of face-down and face-up playing cards (individually and in a deck)
Requirements
- The robot must be able to recognize the playing cards by suit and rank.
- The player’s played cards must be recognizable by suit and rank.
- Duplicate recognition of the same playing card must be avoided.
The cards were visually recognized using a camera connected to the Raspberry Pi via a ribbon cable. In the first implementation phase, the playing cards were recognized in the camera image based on QR codes. In a second phase, a neural network was trained to recognize the cards.
Implementation – Variant 1: QR Codes
Each playing card was marked with two QR codes, one on the top edge and one on the bottom edge. These codes contain information about the suit and the trump suit of the playing card.
The QR codes are recognized using a Python script. This script uses a library to extract the information from the QR codes and then transmits it to the other system components.
Implementation – Variant 2: Neural Network
In a second phase of development, a neural network was trained using images of the cards so that the cards can be recognized without having to alter them. This variant is currently still being tested and integrated into the overall system.

Subproject: ROBOT CONTROL
Task
- Design and Construction of the Play Area
- Selection and control of a robot system and gripper
Module Description
The purpose of this system component is to transport cards using a robotic arm. The key requirement is the smooth movement of the robotic arm. With the 6 axes of the Niryo One robotic arm, it is possible to achieve gameplay that closely mimics that of a real hand.
In addition, the robotic arm serves as a camera mount for the “Card Recognition” subproject. Depending on current needs, the system switches between the “vacuum gripper” tool and the “camera” tool by rotating a custom-designed mount for the gripper and camera.
Movements implemented as part of this subproject include:
- Picking up cards from a face-down stack and placing them in one of the five card holders
- Dealing a card from one of the five card holders onto the table
- Scanning with the camera to detect playing cards in the card holders
- Positioning the camera above the table to detect played cards
- Calibration run
Implementation
- Picking up the cards using a suction cup
- Design and 3D printing of all necessary mechanical components (card holder, card tray, tool holder, hose/cable routing)
- Programming the robot paths in Python
- Invoking robot paths in response to requests from other system components via MQTT (central game control, card recognition)
- Use of the Python library Paho for MQTT interactions
We used ...
Niryo One robot, ROS, vacuum suction cup, MQTT, Raspberry Pi 3B+, Raspi camera, Python, eSpeak, SNIPS, AI, QR codes, 3D printing, YOLO, ...
Impressions
Uses
The video shows how the card-playing robot recognizes cards, responds to voice commands, and moves them with its robotic arm.


















