Sign Language Translator

A Flutter app for translating submitted signing media into text, with device speech and sharing, authenticated history, and a Gemini-powered Python backend.

The frontend manages accounts, media selection, camera recording, results, and history. The backend verifies Firebase ID tokens, validates uploads, requests American Sign Language interpretation into English from Gemini, validates structured results, and saves successful translations under the authenticated user's ID.

Status: backend integration verified locally; Android/web configuration supplied. After the folder reorganization, 113 backend tests passed on 8 October 2026, dependencies were consistent, and temporary Waitress startup/readiness checks passed. Live Gemini/Firebase access and SQLite integrity were last checked on 6 October. Device testing, signed-in media inference, Flutter builds, and recognition accuracy still require validation. See verification evidence.

Credentials on a fresh clone: .gitignore excludes backend .env, Firebase Admin JSON, local Firebase client configuration, private/signing keys, saved sessions, logs, and databases. Configure backend/.env from .env.example, supply your own Admin credentials, and generate frontend/lib/firebase_options.dart plus frontend/android/app/google-services.json for your Firebase project before building the app. See the frontend setup. Existing local credentials are preserved.

Go to

• Features
• Architecture
• Technology stack
• Quick start
• Usage
• Configuration
• API overview
• Database and logging
• Project structure
• Verification and statistics
• Deployment
• Troubleshooting
• Documentation
• Contributing
• Current limitations and next steps
• License

Features

Area	Implemented behavior
Accounts	Email/password registration, sign-in, password reset, sign-out, and profile-name updates through Firebase
Upload	Authenticated image/video submission with format and size checks
Camera	Record a clip, stop recording, then submit it for processing
Results	Displays translation text with device speech/share controls; the API also returns word/clip metadata and model estimates
Speech and sharing	Device text-to-speech, text copying/sharing, and platform-dependent audio export
History	User-scoped persistence, newest-first pagination, Load More, refresh, and clear
Operations	Readiness, structured errors, request IDs, bounded inference concurrency, cleanup, and rotating logs

Recognition processes submitted media. The camera preview does not continuously stream translations. Several files in one API batch are independent inputs, rather than a temporal image sequence.

Architecture

flowchart LR
    User[User] --> App[Flutter app]
    App -->|Sign-in and ID token| Auth[Firebase Authentication]
    App -->|Own profile| Profiles[Firestore users/uid]
    App -->|Bearer token and multipart media| API[Flask API / Waitress]
    API -->|Verify identity| Admin[Firebase Admin SDK]
    Admin --> Auth
    API -->|Validate and stage media| Temp[Temporary request files]
    API -->|Upload and interpret ASL| Gemini[Gemini API]
    API -->|Successful results by UID| SQL[(SQLite history)]
    API -->|Safe operational records| Logs[Rotating logs]
    API -->|Text and metadata| App
    App --> Speech[Device TTS and sharing]

Loading

The app uses Firebase directly for accounts/profiles. Media inference and translation history go through the Python API. Private Gemini and Firebase Admin credentials remain in the backend environment. Uploaded media is temporary; SQLite stores translation metadata. Provider-file deletion is attempted during cleanup.

See architecture and data flows for sequence diagrams, the storage schema, error paths, and concurrency.

Technology stack

Layer	Technology and purpose
Client	Flutter/Dart, camera/file selection, HTTP, device TTS, and native sharing
API	Python, Flask, Flask-CORS, and Waitress
Identity	Firebase Authentication and Firebase Admin token verification
Profiles	Cloud Firestore with owner-scoped rules
Recognition	Google GenAI SDK with aiohttp; Pydantic validation of structured Gemini results
History	Python's embedded SQLite with transactions, parameterized queries, and WAL mode
Configuration	Environment variables and python-dotenv
Operations	Rotating UTF-8 application logs and JSON request/timing records
Testing	pytest and Flutter HTTP-contract tests

The updated frontend lockfile requires Dart >=3.11.0-0 <4.0.0 and Flutter >=3.29.0. Choose a Flutter SDK whose bundled Dart satisfies that requirement; the source's broad Dart constraint starts at 3.0. Runtime package ranges are in backend requirements; verified installed versions are in validation.

Quick start

1. Prepare the environment

You need Python 3.10+, a Flutter SDK with Dart 3.11+, an Android device/emulator or Chrome, a Firebase project, Firebase Admin credentials, and a Gemini API key with access to the configured model.

Clone or download this repository and open the project root containing frontend/, backend/, and this README. If starting from the supplied outer signLangApp_flutter-FYP workspace:

cd .\signLangApp_flutter-master

Backend commands run from this project root. Flutter commands run inside frontend/, where pubspec.yaml is located.

2. Install backend dependencies

.\backend\setup.ps1

Setup creates backend/.venv, installs runtime requirements and pytest, and creates backend/.env if missing. Existing environment values are preserved. Backend setup also provides manual venv, activation, and Linux/macOS commands.

3. Configure credentials and Firebase

Edit backend/.env:

GEMINI_API_KEY=<your Gemini API key>
GEMINI_MODEL=gemini-3.8-flash
FIREBASE_PROJECT_ID=projectdmkflutter
GOOGLE_APPLICATION_CREDENTIALS=./firebase-admin.json
HOST=0.0.0.0
PORT=5000

The relative credential path refers to backend/firebase-admin.json; an absolute path is also supported. Local environment files, private credentials, databases, logs, and virtual environments are excluded by .gitignore.

The supplied Android/web Firebase settings in frontend/ reference projectdmkflutter. Use matching server credentials, or replace the client/server settings for your own project. Enable Email/Password Authentication, provision Firestore, and review/apply firestore.rules. frontend/firebase.json references that shared policy as ../firestore.rules. Backend setup does not deploy those rules.

4. Start and check the backend

.\backend\start.ps1

The script uses the virtual-environment Python, enters backend/, and runs Waitress on 0.0.0.0:5000. Leave the terminal open; Ctrl+C stops it. In a second terminal:

Invoke-RestMethod http://localhost:5000/health
Invoke-RestMethod http://localhost:5000/ready

/ also returns health JSON. /health answers HTTP 200 with configuration flags. /ready returns 503 when required configuration is unavailable; it does not test current provider quota or perform recognition.

5. Run the Flutter app

In a separate terminal, enter the frontend package from the project root:

cd .\frontend
flutter pub get

Run the following Flutter commands from frontend/. The backend stays running in its own terminal. Use a terminal at the project root for backend and firewall commands.

Development target	Backend address
Physical Android phone	http://192.168.100.6:5000 — supplied computer Wi-Fi IP
Android emulator	http://10.0.2.2:5000 — emulator access to its host
Web on the same computer	http://localhost:5000 — exact frontend origin must be allowed

Physical phone on the same reachable LAN:

flutter run --dart-define=BACKEND_URL=http://192.168.100.6:5000

Android emulator:

flutter run --dart-define=BACKEND_URL=http://10.0.2.2:5000

For web, set CORS_ORIGINS=http://localhost:8080 in backend/.env, restart the backend, and run:

flutter run -d chrome --web-hostname=localhost --web-port=8080 --dart-define=BACKEND_URL=http://localhost:5000

The Android default uses 192.168.100.6; override BACKEND_URL when the computer's IP changes. VMware adapter addresses belong to virtual-machine networks. If Windows Firewall blocks the phone, run this from an Administrator PowerShell in the project directory:

.\backend\enable-lan.ps1

The helper adds a scoped TCP 5000 rule for the configured Wi-Fi interface and local subnet. Check http://192.168.100.6:5000/health in the phone's browser. Local HTTP is enabled for Android debug development; release use requires HTTPS.

Usage

1. Register or sign in. Accounts and password reset use Firebase.
2. Choose media or record a clip. Allow relevant permissions and stop recording before submission.
3. Submit for translation. The app sends media with the current user's Firebase ID token.
4. Review the result. Read the translated text and use speech, copying, or sharing controls.
5. Open history. Refresh recent records, load older translations, or clear your saved history.
6. Update your profile or sign out through account controls.

flowchart LR
    SignIn[Sign in] --> Choice{Input}
    Choice --> Upload[Select media]
    Choice --> Camera[Record and stop]
    Upload --> Submit[Submit translation]
    Camera --> Submit
    Submit --> Result[Review result]
    Result --> Speak[Speak / copy / share]
    Result --> History[Saved history]

Loading

Audio export depends on platform/device support; its implementation is limited to Android/iOS, while this repository has no ready iOS application. Model confidence is an estimate, and no independently measured recognition accuracy is supplied.

See the complete usage guide for account flows, media processing, device setup, and recovery from errors.

Configuration

.env.example lists backend settings. Process environment variables take precedence over .env. Relative credential/data/log paths resolve from backend/.

Setting	Default / purpose
GEMINI_API_KEY	Required private API key
GEMINI_MODEL	gemini-3.8-flash
GEMINI_TIMEOUT_SECONDS	180 per file for upload, preparation, and generation
FIREBASE_PROJECT_ID	projectdmkflutter; must match the client
GOOGLE_APPLICATION_CREDENTIALS	Service-account path, or unset for Application Default Credentials
FIREBASE_CHECK_REVOKED	true
HOST, PORT	0.0.0.0, 5000
DATA_DIR	data; database and temporary upload directories
LOG_DIR, LOG_LEVEL	logs, INFO
LOG_MAX_BYTES, LOG_BACKUP_COUNT	5242880 bytes, 3 backups per file
MAX_UPLOAD_MB	100 MiB per file
MAX_FILES_PER_REQUEST	4
MAX_CONCURRENT_PREDICTIONS	2; excess requests receive 429
CORS_ORIGINS	Comma-separated exact web origins

The default total multipart body limit is 101 MiB, including framing, so four maximum-size files cannot fit in one request. The frontend submits one selected file or recording at a time. See backend configuration for storage and deployment details.

API overview

Method	Endpoint	Purpose	Authentication
GET	/, /health	Liveness and configuration flags	Public
GET	/ready	Configuration readiness	Public
POST	/predict, /translate	Multipart recognition	Firebase ID token
GET	/history?limit=100&offset=0	User translations, newest first	Firebase ID token
DELETE	/history	Clear user history	Firebase ID token

Authenticated requests use Authorization: Bearer <Firebase ID token>. The server derives the UID from that token. Account/profile operations use Firebase directly.

Submit media under exactly one field: file, video, or repeated files. Supported extensions are PNG, JPG/JPEG, WEBP, MP4, M4V, MOV, AVI, WEBM, MPEG, and MPG, subject to file-content validation and provider processing.

Example after obtaining a Firebase ID token through a signed-in client and setting it in the terminal environment:

curl.exe --request POST "http://localhost:5000/predict" `
  --header "Authorization: Bearer $env:FIREBASE_ID_TOKEN" `
  --form "file=@C:\media\sign.mp4"

Prediction returns results and formats; history returns a JSON array. Results contain text aliases, an ID, timestamps, words, clip metadata, and model-estimated confidence. Errors contain error, code, and request_id. Every response includes X-Request-ID.

The API reference covers full schemas, example responses, pagination, errors, and status codes.

Database and logging

Storage

Data	Storage	Behavior
Accounts	Firebase Authentication	Sign-in, password reset, ID tokens
Profiles	Firestore users/{uid}	Name, email, creation time; owner rules
Translation history	SQLite backend/data/history.sqlite3	UID-scoped results; JSON metadata in a SQL text column
Media	Request-specific temporary files	Local cleanup on exit; remote provider deletion attempted

SQLite is an embedded SQL database included with Python, requiring no separate database server. History queries are parameterized, writes are transactional, and WAL mode is enabled. A batch is saved only after all items succeed. Preserve DATA_DIR and use SQLite-aware backups for active databases. Independent backend replicas on multiple machines need shared database infrastructure.

Logs and error handling

Default file	Contents
backend/logs/app.log	Startup, authentication/provider warnings, internal error locations
backend/logs/requests.log	JSON request ID, route, method, status, error code, duration
backend/logs/performance.log	JSON total-request and Gemini translation timings

Files append as requests arrive, rotate at 5 MiB, and retain three backups each. UTC timestamps/request IDs connect failures to operational records. Bodies, tokens, credentials, and translation text are excluded.

Python try/except/finally and Flask handlers distinguish expected failures from internal errors and release temporary files/concurrency slots. Internal errors return sanitized HTTP 500 responses. Provider cleanup is bounded and best effort. Logs have no public download endpoint.

Get-Content .\backend\logs\requests.log -Tail 20 -Wait

Ctrl+C stops watching. Regular backend tests disable file logging; logging tests explicitly enable it.

Project structure

.
├── README.md                         Project guide and navigation
├── CONTRIBUTING.md                   Development/review workflow
├── frontend/
│   ├── README.md                     Flutter setup and frontend commands
│   ├── pubspec.yaml / pubspec.lock   Flutter dependencies
│   ├── lib/
│   │   ├── main.dart                 Initialization and navigation
│   │   ├── firebase_options.dart     Local Firebase client configuration (ignored)
│   │   ├── screens/                  Accounts, upload, camera, results, history
│   │   ├── services/                 Firebase auth, HTTP, permissions
│   │   ├── utils/                    User-feedback helpers
│   │   └── widgets/                  Shared UI
│   ├── test/                         Flutter contract tests
│   ├── assets/                       Fonts, audio, processing animation
│   ├── android/                      Runner, permissions, debug HTTP
│   ├── web/                          Web runner
│   ├── linux/                        Runner; Firebase initialization unsupported
│   └── firebase.json                 FlutterFire settings and shared rules reference
├── backend/
│   ├── app.py                        Routes, tracing, error handlers
│   ├── config.py                     Environment, paths, limits
│   ├── firebase_service.py           Firebase ID-token validation
│   ├── gemini_service.py             Async recognition/result validation
│   ├── transformer_service.py        Compatibility entry points
│   ├── history_store.py              Transactional SQLite history
│   ├── media.py                      Upload validation/temp files
│   ├── logging_setup.py              Rotating UTF-8 logs
│   ├── server.py                     Waitress entry point
│   ├── setup.ps1 / start.ps1          Windows setup and foreground start
│   ├── enable-lan.ps1                Scoped Wi-Fi firewall helper
│   ├── smoke_check.py                WSGI HTTP checks
│   ├── verify_setup.py               Configuration/credential checks
│   ├── requirements*.txt             Runtime/test requirements
│   ├── pytest.ini                    Backend test discovery and local cache
│   ├── .env.example                  Configuration template
│   ├── tests/                        Backend regression/logging tests
│   ├── data/                         Generated database/temp state; ignored
│   └── logs/                         Generated logs; ignored
├── docs/                             Architecture, usage, API, statistics, evidence
├── output/pdf/                       Generated project report
└── firestore.rules                   Owner-scoped profile rules

The frontend and backend keep separate dependencies and entry points. This root README covers the whole project; each application folder has its own setup guide. The backend uses a local Python virtual environment and Waitress. Docker configuration is not included.

Verification and statistics

Source inventory recorded on 4 October 2026; backend tests, dependency consistency, and temporary Waitress serving were rechecked after reorganization on 8 October 2026. Live provider credentials and SQLite integrity retain their 6 October verification date. These figures describe local observations; they are not hosted CI, coverage, accuracy, or throughput benchmarks.

Metric / check	Recorded result
Scoped source inventory	36 files / 5,049 physical lines; 35 / 4,978 excluding generated Firebase options
Backend suite	113 passed in 5.09 seconds on 8 October; no cache warnings
Dependencies	Required imports/version ranges valid; pip check found no conflicts
Waitress startup	Temporary local smoke server: /health and /ready 200; unauthenticated /history 401
Normal server operation	The smoke checker stops its temporary server; run backend/start.ps1 to serve the app
Logging	Rotation/correlation tests passed on 8 October; runtime file updates were recorded on 6 October
SQLite	Integrity check passed and WAL mode confirmed on 6 October
Gemini configuration	Model metadata access verified on 6 October without content generation
Firebase configuration	OAuth credential refresh verified on 6 October without user-record reads
Flutter	11 HTTP-contract cases supplied; execution/build/device verification pending
Firestore rules	Configuration supplied; rule deployment/emulator verification pending

Backend checks:

.\backend\.venv\Scripts\python.exe -m pip check
.\backend\.venv\Scripts\python.exe -m pytest backend\tests -q
.\backend\.venv\Scripts\python.exe backend\smoke_check.py

Optional live configuration checks without media upload/content generation:

.\backend\.venv\Scripts\python.exe backend\verify_setup.py --live

With a configured Flutter SDK, starting from the project root:

cd .\frontend
flutter analyze
flutter test
cd ..

Backend tests inject service doubles and temporary storage. Signed-in media submissions and device checks remain necessary. Project statistics defines the counted scope, dependencies, routes, and limits; validation records package versions and test evidence.

Deployment

The backend runs directly through Waitress on Windows/Linux using its Python virtual environment. The backend guide provides setup, foreground startup, configuration, and persistent SQLite storage instructions.

For deployment, configure HTTPS, exact web origins, private backend credentials, persistent writable data/log directories, and aligned Firebase project settings/rules. Android releases also need signing configuration and an HTTPS backend URL. Multiple independent backend machines require a shared database design.

No public deployment is included in the verification record.

Troubleshooting

Symptom	Check or action
Phone cannot connect	Keep server running; check current Wi-Fi IP/reachability and the LAN firewall helper
HTTP 404	Browser checks use /, /health, /ready; recognition requires multipart POST /predict
Translation/history returns 401	Sign in and send a current ID token for the matching Firebase project
/ready returns 503	Check .env, service-account path, model configuration, and app.log
Web CORS error	Match scheme/host/port in CORS_ORIGINS, then restart
Upload rejected	Check format/signature, nonempty content, per-file and total request size
Inference returns 429/502/504	Match request ID to logs; check capacity, provider availability/quota, and timeout
Profile update fails	Check Firestore provisioning and deployed owner rules
Logs absent	Start backend and ensure LOG_DIR is writable; regular tests disable file logging
History lost after replacement	Preserve the backend's DATA_DIR and SQLite database
Camera/TTS/sharing differs	Check actual device permissions and platform behavior

See usage and backend troubleshooting for details.

Documentation

Guide	Go to it for
Architecture	Components, sequences, schema, identity, and cleanup
Usage	Setup, account/media flows, device addresses, troubleshooting
API reference	Authentication, multipart fields, responses, errors, pagination
Frontend guide	Flutter package setup, device addresses, and frontend checks
Backend guide	Windows/Linux/manual setup, configuration, logs, Waitress
Project statistics	Source counts, versions, dependencies, configured limits
Validation	Recorded checks and unverified areas
Project analysis	Integration findings, fixes, and design constraints
Contributing	Development checks and pull-request expectations

Contributing

Follow CONTRIBUTING.md. Preserve authenticated user isolation and the API contract; update the relevant guides when behavior changes. Record checks actually run and their limitations.

Current limitations and next steps

• Evaluation: no local trained ASL checkpoint, labeled evaluation dataset, measured recognition accuracy, or calibrated confidence is supplied.
• End-to-end checks: validate signed-in upload, two-user history isolation, camera, profile/password reset, speech/audio sharing, and provider failures with real services.
• Platforms: Android/web are configured but unbuilt here. Windows/Linux Firebase initialization is unsupported. iOS/macOS Firebase values are placeholders and their app projects are absent.
• Release operations: deploy/test Firestore rules, configure HTTPS/signing and server operation, and verify persistence/backups.
• Scaling: introduce shared database infrastructure before independent backend replicas on multiple machines.

License

No LICENSE file is currently included. The project owner should specify a license before distributing the repository with reuse permissions.