Card-Playing Robot

Many people enjoy playing cards, but sometimes there isn't enough of a player!

10/01/2019 - 03/14/2020

Card-Playing Robot

Project Description

Card games are a popular pastime for people of all ages. Depending on the card game, a specific number or a minimum number of players is required; otherwise, the game cannot be played. Since the required or desired number of players is not always available, the goal of Systems Engineering 1 is to develop a robotic system that can replace a human player for playing various card games with real playing cards.

People Involved

Student Group: 

Central Game Control: Markus Wagner, Marco Westermair, Dominik Ziegler

Human-Machine Interaction: Tobias Ertle, Manuel Mikschl, Alexander Markidonow

Card Recognition: Benedikt Degenstein, Ali Gözen, Thomas Henning, Dennis Religa, Burak Saglam

Robot Control: Michael Durach, Thomas Höger, Jan Sigg, Nikolas Voss

Advisors: Prof. Dr.-Ing. Martin Bayer, Prof. Dr.-Ing. Claudia Meitinger

Course

Systems Engineering 1 (E5-IK)

Duration

Winter Semester 2019/2020

Objective

Development of a functional prototype of a card-playing robot,

  • which can replace a human player in the Bavarian card game "Watten" for two players.
  • that can play with real playing cards.
  • that can be easily expanded or modified (e.g., to support other card games, other technologies for card recognition and/or card handling, etc.).

System Overview

Functionalities to Be Implemented

To achieve the objective, the following technical functionalities should be implemented:

  • Central game control system to manage the other game components
    • Interaction with human players whenever appropriate or necessary
    • Selecting and playing a card when it is the robot player’s turn (at a minimum, in accordance with the rules; if possible, using a simple tactic)
  • Interaction with the human player(s) via various modalities
  • Identifying playing cards
    • Face-down cards—i.e., the robot player’s own cards that have not yet been played and should not be visible to human players
    • face-up cards, i.e., the most recently played card (regardless of which player played it)
  • Picking up and moving the robot
      player’s playing cards
    • from the deck at the start of the game
    • during the game, from a card holder

Arrangement of System Components

The following figure shows the selected player arrangement as well as the defined storage locations for playing cards before and during the game. By OpenPNP itself.

Modularization

In accordance with the technical functionalities to be implemented, the project was carried out as four subprojects. Each functionality was implemented on a Raspberry Pi as part of an MQTT network.

Procedure

The project was implemented in several phases over the course of the semester.

Setting up the project

  • Learning about the Wadden Sea
  • Creating the requirements and specifications
  • Planning the structure, work packages, workflow, project plan, and costs
  • Defining synchronization milestones
  • Initialize Git repositories

Perform a full-system simulation

Objectives:

  • Define a binding interface d (here: MQTT topics)
  • Reach a common understanding of the overall system behavior

Procedure:

  • Each group simulates the behavior of its subsystem.
  • Communication between subsystems/groups is permitted exclusively via MQTT (e.g., using command-line tools).

Integrate the system

  • During each system integration, an attempt is made to run through a complete game sequence.
  • Errors and opportunities for optimization are documented and subsequently addressed by the groups.
  • At the next weekly meeting, the improvements and enhancements are tested again in the form of a complete game run.

Conclude the project

  • Presentation of the results of the subprojects
  • Documentation of the modules
  • Acceptance / Handover

Subproject: CENTRAL GAME CONTROL

Tasks

  • Central control of gameplay
    • Interaction with human players whenever appropriate or necessary
    • Requesting robot actions
    • Requesting map recognition
  • Implementation of game logic
    • Determining which card to play: in accordance with the rules and using a simple game strategy

Module Description

A Python script was developed to control communication among the groups, manage the game flow, and implement the strategy. A systemd service automatically starts the scripts at system startup, eliminating the need for manual intervention.

To ensure flexibility for future changes, an object-oriented design was implemented in which player types (human/robot) can be swapped just as easily as the type of game (Watten/...) or the method of communication between components.

The card-selection function assigns a value to each card, which is taken into account during decision-making. In addition, a simple heuristic is applied when choosing the play and suit, enabling the system to win games.

In addition, an MQTT logger and a simulation program were created for internal testing. The messages sent by all system components were logged to facilitate troubleshooting. The simulation program allows for testing of the team’s own software without relying on other groups.

Subproject: HUMAN-MACHINE INTERACTION

Responsibilities

  • Design of the interaction concept in consultation with the central game control system
  • Implementation of voice input, voice output, and touch interaction

Requirements (selection)

  • The interaction should be self-explanatory for human players, i.e., usable without instruction.
  • The system’s state—in terms of game progress and selected parameters (e.g., shot type, ball color)—should be visible to the player.
  • The robot should react to game-critical actions in a human-like manner.

Implementation

  • Programming language: Python 3
  • Touchscreen display as the
    basic building block, with voice output and voice input as supplementary features
  • Paho-MQTT library for MQTT messages
  • Startup script to start/restart all programs as individual tasks (voice input, voice output, and display)
  • Kivy library for the user interface
  • Text-to-speech using Eclipse
  • Voice input using Snips

Usage

After the Raspberry Pi 3 boots up, the Python startup script must be executed. This script then waits for commands arriving via MQTT.

As soon as user input is required, the player is prompted via voice output and on-screen instructions.

In addition to touch input, this is also possible via voice input.

Subproject: MAP RECOGNITION

Task

  • Identification of face-down and face-up playing cards (individually and in a deck)

 Requirements

  • The robot must be able to recognize the playing cards by suit and rank.
  • The player’s played cards must be recognizable by suit and rank.
  • Duplicate recognition of the same playing card must be avoided.

The cards were visually recognized using a camera connected to the Raspberry Pi via a ribbon cable. In the first implementation phase, the playing cards were recognized in the camera image based on QR codes. In a second phase, a neural network was trained to recognize the cards.

Implementation – Variant 1: QR Codes

Each playing card was marked with two QR codes, one on the top edge and one on the bottom edge. These codes contain information about the suit and the trump suit of the playing card.

The QR codes are recognized using a Python script. This script uses a library to extract the information from the QR codes and then transmits it to the other system components.

Implementation – Variant 2: Neural Network

In a second phase of development, a neural network was trained using images of the cards so that the cards can be recognized without having to alter them. This variant is currently still being tested and integrated into the overall system.

Subproject: ROBOT CONTROL

Task

  • Design and Construction of the Play Area
  • Selection and control of a robot system and gripper

Module Description

The purpose of this system component is to transport cards using a robotic arm. The key requirement is the smooth movement of the robotic arm. With the 6 axes of the Niryo One robotic arm, it is possible to achieve gameplay that closely mimics that of a real hand.

In addition, the robotic arm serves as a camera mount for the “Card Recognition” subproject. Depending on current needs, the system switches between the “vacuum gripper” tool and the “camera” tool by rotating a custom-designed mount for the gripper and camera.

Movements implemented as part of this subproject include:

  • Picking up cards from a face-down stack and placing them in one of the five card holders
  • Dealing a card from one of the five card holders onto the table
  • Scanning with the camera to detect playing cards in the card holders
  • Positioning the camera above the table to detect played cards
  • Calibration run

Implementation

  • Picking up the cards using a suction cup
  • Design and 3D printing of all necessary mechanical components (card holder, card tray, tool holder, hose/cable routing)
  • Programming the robot paths in Python
  • Invoking robot paths in response to requests from other system components via MQTT (central game control, card recognition)
  • Use of the Python library Paho for MQTT interactions

We used ...

Niryo One robot, ROS, vacuum suction cup, MQTT, Raspberry Pi 3B+, Raspi camera, Python, eSpeak, SNIPS, AI, QR codes, 3D printing, YOLO, ...

Uses

The video shows how the card-playing robot recognizes cards, responds to voice commands, and moves them with its robotic arm.

Other Projects at THA

A man wearing a helmet and carrying a leather shoulder bag is riding a bicycle

07/01/2026 - 06/30/2029

A2W

Active to Work - Active Commuting: Effects on Mental Health and the Role of Urban Environmental Factors

Discover project
Students wearing VR headsets

04/01/2026 - 04/30/2029

Gesellschaft • KI

AI-Supported, VR-Based German Language Learning for Future Professionals

The project “AI-Supported, VR-Based Skills Competition for Future Professionals in the Field of German as a Technical and Professional Language” addresses the ongoing shortage of skilled workers

Discover project

02/01/2026 - 01/31/2027

GeWo-Bay Study

An exploratory field study is examining health care for homeless people in Bavaria.

Discover project

02/01/2026 - 01/31/2029

Digitalisierung • Robotik

BOSDra

BOSDra is developing an automated first-response drone system for law enforcement and other organizations with security responsibilities (BOS).

Discover project
Ambulance Image

01/01/2026 - 12/31/2027

WARM Study

Effects of Environmental Factors on Acute and Chronic Wounds, Mental Health Conditions, and Social Factors Among People Experiencing Housing Crises in Augsburg

Discover project
Two field workers use an app on a construction site to install fiber-optic cables

01/01/2026 - 06/30/2027

Digitalisierung

DocPilot

Digital Data Management for Fiber-Optic Network Expansion

Discover project

01/01/2025 - 12/31/2026

Ressourceneffizienz • Wasser

Cost-Effective Upgrades to Wastewater Treatment Plants

Cost-Effective Upgrading of Wastewater Treatment Plants by Improving the Settling Properties of Activated Sludge

Discover project
Cover photo: DiPukS

08/01/2024 - 06/30/2027

Digitalisierung

DiPukS

Digital Planning Support for Climate-Neutral Urban Development

Discover project