Open to Data Engineering Opportunities
Lahore, Pakistan
Owais Ahmad, Data Engineer and AI/ML Engineer

OWAIS AHMAD

Data Engineer | AI/ML Engineer | Software Developer

Building Reliable Data Pipelines & Scalable Lakehouse Solutions

Data Engineer specializing in scalable ETL/ELT pipelines, Lakehouse architectures, data integration, and analytics-ready solutions. Skilled in Python, SQL, PySpark, Databricks, Apache Spark, Delta Lake, PostgreSQL, FastAPI, Docker, and Power BI, with a focus on transforming raw data into reliable, high-quality, and actionable datasets.

PythonSQLPySparkDatabricksApache SparkDelta LakeETL/ELT
Medallion_Pipeline.py
LIVE DATA FLOW
Data Sources
REST APIs | Relational DBs | Files
INGEST
Bronze LayerRAW
Auto Loader / JDBC Ingestion
Append-Only
Silver LayerCLEANED
Deduplicated & Validated (MERGE)
PySpark
Gold LayerCURATED
Business KPIs & Aggregates
Delta Lake
Analytics & BI
Power BI | Databricks SQL | AI-BI
Latency: Real-time / BatchStatus: Healthy
BACKGROUND & PHILOSOPHY

About Me

I am a Data Engineer and AI/ML Engineer specializing in scalable data pipelines, Lakehouse architectures, intelligent automation, and AI-powered applications. I build reliable data platforms, backend services, and machine learning solutions using Python, SQL, PySpark, Databricks, Delta Lake, PostgreSQL, FastAPI, Docker, and Power BI.

My experience spans Bronze-Silver-Gold Medallion Architecture, incremental processing, data validation, REST API development, quality-assurance CRM workflows, and operational reporting. I also build backend integrations and automate business workflows, with a focus on scalable, reliable, and maintainable systems.

With a background in Data Science and AI/ML, I also enjoy building intelligent solutions that combine data engineering, machine learning, and modern software technologies to solve real-world problems.

Lakehouse Architecture Design
PySpark & Distributed Computing
Delta Lake MERGE Operations
Automated Auto Loader Ingestion
FastAPI Backend Development
CRM & REST API Development
Data Integration & Validation
Workflow Automation
Power BI & Operational Dashboards

Medallion Architecture

Structured Bronze-Silver-Gold data layers for clean, audited, and analytics-ready datasets.

Incremental Processing

Efficient Delta MERGE & Auto Loader routines minimizing batch latency and compute overhead.

Data Quality & Governance

Centralized schema validation, deduplication rules, and Unity Catalog access control.

Primary Focus: Reliability & MaintainabilityLahore, PK
DATA PIPELINE ARCHITECTURE

Medallion Lakehouse Architecture

How raw data flows from disparate sources into Bronze, Silver, and Gold Delta Lake layers for analytics & reporting.

Unity Catalog Governed
01SOURCES

Data Sources

Raw transactional data, system logs, and hardware device feeds entering the data platform.

REST APIs
Relational DBs (PostgreSQL/MySQL)
JSON / CSV / Parquet Files
Biometric Records
02INGESTION

Ingestion Layer

Ingestion framework handling file arrival, schema preservation, and network retries.

Databricks Auto Loader
JDBC Connectors
FastAPI Endpoints
Scheduled Data Workflows
03BRONZE

Bronze Layer

Raw data landed directly into Delta Lake tables maintaining auditability without initial transformation.

Raw Delta Lake Tables
Raw Landing Tables
Source Schema Preservation
Ingestion Timestamps
04SILVER

Silver Layer

Cleansed, structured, and validated data tables ready for downstream business logic.

Schema Validation & Profiling
Deduplication Logic
Delta MERGE Upserts
Data Quality Checks
05GOLD

Gold Layer

Curated, aggregated data models optimized for business decision-making and KPI metrics.

Business Rule Processing
Aggregated Business Datasets
Sales & Customer KPIs
Curated Delta Tables
06CONSUMPTION

Analytics & BI

Reporting layer connecting downstream analytics tools directly to trusted data.

Power BI Dashboards
Databricks SQL Dashboards
Databricks AI/BI
REST API Services
Sources: APIs / DBs / FilesAuto Loader / JDBC IngestionBronze (Raw Delta)Silver (Validated Delta)Gold (Business Aggregates)Power BI & Databricks SQL
CAREER & INDUSTRY EXPERIENCE

Work Experience

Unified Marketing Solutions (UMSBPO)•Current Role

Data Engineer & Developer

October 2026 - Present
Islamabad, Pakistan

Developing quality-assurance CRM capabilities and backend services while integrating, validating, and automating operational data workflows.

Key Engineering Responsibilities:

Develop quality-assurance CRM features for operational workflows.
Build backend services using Python and FastAPI.
Manage PostgreSQL data storage and queries.
Integrate customized VICIdial APIs and synchronize completed-call data.
Validate, process, and integrate operational data.
Automate business workflows.
Build operational dashboards and reporting.
Technologies used:PythonFastAPIPostgreSQLSQLDockerREST APIsCRM DevelopmentVICIdial APIData IntegrationETL/ELTWorkflow Automation
YBIX•Previous Role

Data Engineer

May 2026 - October 2026
Lahore, Pakistan

Designing, building, and maintaining production-grade Lakehouse architectures and data pipelines for scalable analytics.

Key Engineering Responsibilities:

Build and maintain production-grade ETL/ELT pipelines using Python, PySpark, Databricks, and Spark SQL.
Design Bronze, Silver, and Gold Lakehouse layers using Delta Lake and Medallion Architecture.
Perform data profiling, cleansing, transformation, validation, and quality checks.
Implement Delta Lake MERGE operations for incremental loading and upserts.
Create analytical datasets, KPI dashboards, and business reports.
Develop reusable, modular, and maintainable Data Engineering workflows.
Document pipeline architecture and data flows.
Collaborate with analysts and business stakeholders to translate requirements into scalable data solutions.
Technologies used:PythonPySparkApache SparkDatabricksDelta LakeSpark SQLMedallion ArchitectureETL/ELTData ProfilingPower BI
January 2023 - January 2025
Abu Dhabi, UAE

Worked as a Data Entry Operator at Oasis House General Contracting Co. L.L.C. in Abu Dhabi, UAE. Employment was from January 2023 to January 2025.

EMPLOYMENT DETAILS:

Data Entry Operator role
Employment period: 10 January 2023 - 9 January 2025
Location: Abu Dhabi, UAE
Experience letter available
TECHNICAL STACK & COMPETENCIES

Technical Skills

Technologies, frameworks, and tools for data engineering, backend development, integrations, and Lakehouse architectures.

Programming

PythonSQL

Data Engineering

ETL/ELTData IngestionData IntegrationData CleaningData TransformationData ValidationIncremental ProcessingData QualityBatch ProcessingWorkflow Automation

Big Data & Lakehouse

PySparkApache SparkDatabricksAzure DatabricksSpark SQLDelta LakeMedallion ArchitectureAuto LoaderDelta MERGEUnity Catalog

Cloud

Azure DatabricksAzure Data Lake

Databases & Warehousing

PostgreSQLMySQLSQLiteData WarehousingDatabricks SQLSpark SQL

BI & Analytics

Power BIDatabricks SQL DashboardsDatabricks AI/BI

APIs & Integration

FastAPIREST APIsCRM DevelopmentVICIdial APIOAuth 2.0LinkedIn APIOdoo APIJDBC

DevOps & Orchestration

DockerDatabricks WorkflowsScheduled JobsRetry Handling

Tools

GitGitHubVS CodeJupyter NotebookStreamlitMicrosoft Excel
FEATURED ENGINEERING WORK

Featured Projects

Data Engineering projects, Databricks workflows, PySpark processing, and REST API integrations.

Databricks medallion architecture showing data sources flowing through Auto Loader, Bronze, Silver, Gold and analytics
LakehouseFEATURED

End-to-End Data Engineering & AI Pipeline

Metadata-driven Lakehouse automation with Databricks, Delta Lake, & Unity Catalog

Built an automated end-to-end Data Engineering pipeline using Databricks, PySpark, Delta Lake, and Medallion Architecture.

ARCHITECTURE FLOWACTIVE
Data Sources → Auto Loader / JDBC → Bronze → Silver → Gold → Databricks SQL / AI-BI
Automated ingestion using Auto Loader and JDBC connectors
Bronze, Silver, and Gold Delta Lake tables implementation
Automated data cleansing, schema validation, and deduplication
DatabricksPySparkSQLDelta LakeAuto Loader+4 more
Case StudyView Details
FMCG data pipeline showing sales, product and customer data processed through Bronze, Silver and Gold layers
LakehouseFEATURED

FMCG Data Engineering & Analytics Pipeline

Scalable sales, product, and customer analytics platform for FMCG data

End-to-end FMCG data engineering pipeline transforming raw sales and inventory records into Gold-layer analytics.

ARCHITECTURE FLOWACTIVE
Raw FMCG Data → Bronze → Silver → Gold → Sales Analytics
End-to-end FMCG data pipeline architecture
Cleansing and schema validation routines
Data transformation and duplicate removal logic
DatabricksPySparkSpark SQLDelta LakeMedallion Architecture
Case StudyView Details
Online retail analytics pipeline from Azure Databricks Lakehouse to Power BI
LakehouseFEATURED

Online Retail Lakehouse & Analytics Pipeline

Transactional analytics processing over 540K records with Power BI

541,909
Raw Transactions Ingested
527,390
Validated Records Cleaned

Processed 541,909 raw retail transactions to yield 527,390 valid records, powering Gold-layer sales KPIs and Power BI dashboards.

ARCHITECTURE FLOWACTIVE
Raw Retail Transactions → Azure Databricks → Bronze → Silver → Gold → Power BI
Processed approximately 541,909 raw retail transactions
Produced 527,390 valid records after automated cleansing and filtering
Removed duplicate transactions and filtered out cancellation orders
Azure DatabricksPySparkSQLSpark SQLDelta Lake+1 more
Case StudyView Details
Automated ETL pipeline showing extraction, PySpark transformation, validation and Databricks processing
ETL/ELT

Automated ETL Pipeline

Batch processing framework with data quality checks & monitoring

Automated extraction, transformation, validation, loading, and monitoring system for distributed batch data processing.

ARCHITECTURE FLOWACTIVE
Source → Extraction → Validation & Transformation → Loading → Monitoring
Automated extraction from diverse data sources
Scalable PySpark transformation engine
Data quality checks and validation rules
PythonSQLPySparkDatabricks
Case StudyView Details
Biometric attendance integration pipeline from biometric device through Python ETL and PostgreSQL to Odoo ERP
API & IntegrationFEATURED

Anviz Attendance Data Integration Pipeline

Biometric attendance sync pipeline linking hardware API to PostgreSQL & Odoo ERP

Automated biometric punch-record ingestion with validation, employee ID mapping, duplicate prevention, and Odoo ERP sync.

ARCHITECTURE FLOWACTIVE
Anviz Biometric Device → API → Python ETL → Validation → PostgreSQL → Odoo ERP
Automated biometric punch-record ingestion from Anviz hardware
Data transformation and validation rules
Relational persistence using PostgreSQL
PythonFastAPIPostgreSQLDockerOdoo API+1 more
Case StudyView Details
LinkedIn connection tracker architecture using OAuth 2.0, FastAPI and PostgreSQL
API & Integration

LinkedIn Connection Tracker

REST API & OAuth 2.0 application tracking connection lifecycle states

FastAPI REST service with OAuth 2.0 tracking LinkedIn connection lifecycles across Pending, Accepted, Rejected, and Withdrawn states.

ARCHITECTURE FLOWACTIVE
LinkedIn API → OAuth 2.0 → FastAPI Backend → PostgreSQL → Analytics
OAuth 2.0 authentication flow
LinkedIn API integration for network data
Connection lifecycle state tracking (Pending, Accepted, Rejected, Withdrawn)
PythonFastAPILinkedIn APIOAuth 2.0PostgreSQL+1 more
Case StudyView Details
Illustrative knee X-ray analysis workflow for KL0 to KL4 osteoarthritis classification
AI & AutomationFEATURED

AI-Based Knee Osteoarthritis Detection

Deep Learning and Vision Transformer-Based Medical Image Analysis

88%
ViT Top-1 Accuracy
100%
ViT Top-5 Accuracy

Academic AI system for analyzing knee X-rays and classifying osteoarthritis severity across KL0-KL4 using deep learning and explainability.

ARCHITECTURE FLOWACTIVE
X-ray → Preprocessing → AI Models → KL Classification → Grad-CAM → API → Android
88% Vision Transformer Top-1 Accuracy
Five-Level Kellgren-Lawrence Classification
Explainable AI with Grad-CAM
PythonDeep LearningCNNResNet50Vision Transformer+7 more
Illustrative QA CRM workflow connecting VICIdial data, FastAPI, PostgreSQL and a QA dashboard
API & IntegrationFEATURED

QA CRM & Quality Management System

Enterprise QA Workflow Automation and CRM Integration

Active-development quality assurance CRM for call evaluations, workflow visibility, structured data, reporting, and VICIdial integration.

ARCHITECTURE FLOWACTIVE
Call Data → API Sync → Validation → PostgreSQL → QA Dashboard → Reporting
Multi-Module QA Management
FastAPI & PostgreSQL
VICIdial API Integration
PythonFastAPIPostgreSQLDockerREST APIs+5 more
Case StudyView Details
Illustrative AI data scientist agent workflow from dataset profiling to reports
AI & AutomationFEATURED

AI Data Scientist Agent

Intelligent Data Analysis and AI-Powered Analytical Automation

An AI-assisted system for data profiling, analytical workflows, insight generation, and automated reporting.

ARCHITECTURE FLOWACTIVE
Dataset → Profiling → Analysis → AI Assistance → Visualization → Report
Automated Data Analysis
AI-Assisted Insights
Intelligent Reporting
PythonPandasNumPyLLM/AI IntegrationData Visualization+3 more
AI personal assistant architecture connecting Streamlit interface with FastAPI and AI processing
AI & Automation

AI Personal Assistant

FastAPI & Streamlit assistant application for workflow automation

AI-powered personal assistant application using FastAPI REST endpoints and Streamlit for workflow automation.

ARCHITECTURE FLOWACTIVE
Streamlit UI → FastAPI Endpoints → Automation Engine
FastAPI REST endpoint architecture
Interactive Streamlit user interface
Workflow automation and task processing
PythonFastAPIStreamlit
Case StudyView Details
PROFESSIONAL CREDENTIALS

Certifications & Training

Technical training and course credentials across Databricks Academy, IBM, and regional IT boards.

KPITB & UET Mardan

Artificial Intelligence & Machine Learning

Issued: June 2026
ID: 144 Training Hours
Machine LearningArtificial IntelligencePython
Verified Credential
Databricks Academy / Simplilearn

Get Started with Databricks for Data Engineering

Issued: July 2026
ID: 10490100
DatabricksPySparkDelta Lake
Verified Credential
Databricks Academy / Simplilearn

Training for Snowflake

Issued: August 2026
ID: 10603969
DatabricksSnowflakeData Lakehouse
Verified Credential
IBM / Coursera

Introduction to Data Engineering

Issued: February 2025
ID: UKQRSDQUEUUA
Data Engineering FundamentalsETL PipelinesRelational Databases
Verified Credential
DigiSkills Training Program

Data Analytics and Business Intelligence

Issued: February 2024
ID: PBQRG7QMK
Data AnalysisBusiness IntelligenceReporting
Verified Credential
Simplilearn SkillUp

Introduction to Data Mining Course

Issued: December 2024
ID: 7647754
Data MiningPattern ExtractionFeature Processing
Verified Credential
IBM / Coursera

BI Dashboards with IBM Cognos Analytics and Google Looker

Issued: February 2025
ID: 5TQDTAHDYCP4
Cognos AnalyticsGoogle LookerDashboard Design
Verified Credential
IBM / Coursera

Get Started with Data Engineering

Issued: IBM Training
Data ArchitectureData Pipeline ConceptsCloud Storage
Training Course
ACADEMIC BACKGROUND

Education

Specialization: Data Science

Bachelor of Science in Computer Science

University of Engineering & Technology (UET) Mardan
Mardan, PakistanGraduation: 2026
Academic StandingCGPA: 3.00 / 4.00
GET IN TOUCH

Let's Build Reliable Data Systems

I'm open to Data Engineering opportunities, collaborations, and projects involving scalable data pipelines, Lakehouse platforms, analytics infrastructure, and API integrations.

Send a Message