Umar Khan

Machine Learning Engineer and Researcher at DFKI, working on Computer Vision and NLP based systems with Dr. Sheraz Ahmed. My previous works include working with Dr.Faisal Shafait on various R&D and commercial Projects under NCAI, since my undergrad I am interested in machines and people alike. I am an avid reader of Psychology and immensely interested in building the future of systems inspired from Biomimicry.

with keen focus on application of research I have worked on 100+ projects with most reaching to production. you can learn more about my fiverr and upwork links in the about section here

Interests: Computer Vision, Graph Neural Networks, Autonomous Systems, Biomimicry.

Publications

• Zeneval: Framework for evaluating agents in custom repositories, U.Khan, Muhammad Asad Ali [coming soon] (Currently Used in production)
• MoTE: Mixture of Task Experts for Multi-Task Video Understanding Muhammad Asad Ali, U.Khan, Nadia Robertini, Didier Stricker [paper] (Currently Used in production, Accepted at BMVC 2026)

• WordVIS: A Color Worth A Thousand Words U.Khan, S.Saifullah, S.Agne, A.Dengel, S.Ahmed. [paper] (Currently Used in production, Preprint Only)

• TabAug: Data Driven Augmentation for Enhanced Table Structure Recognition Umar Khan, Sohaib Zahid, Muhammad Asad Ali, Adnan ul Hassan, Faisal Shafait [paper] Publication Status: Accepted at ICDAR2021

Notable Projects

• GLMEXtract VLM For Extracting Target Centric data from 0.9B GLMOCR Model Replacing Large Pipelines with a VLM allows us to extract target Key Value Pairs from images using a relatively small model pretrained on extractig text from images.m this served as the perfect model. in addition to serving this purpose we came up with a new way of structuring multiple models tasks inside a single model, which was submitted as a publication currently under review.
• MedAgent System A Medical Multi-Agent System for Medical Document Processing We developed a multi-agent system for medical document processing. This system allows our users to process medical billings in large medical cases spanning over 700 pages in a single case. this requires the agent to perform long horizon tasks with no human oversight, interrupts were placed and controlled by system administrators to enable and disable at will.
• Zeneval An Automated Evaluation System for Coding Agents In my personal time I working on a project that allows us to automate evaluations on coding agents and models. both harness specific and independent evaluations. This project will be Open Source and will be available on GitHub. Link

• MemoryCore Developing Memory infrastructure for LLM's Developing Memory infrastructure for LLM's. This currently its a side project that is being offered as a service in both as a chat form as well as an MCP to be added to various LLM clients. You can access the first version of the memory infrastructure in the link below. Chat Now
• DocumentStruct Document Structure Extraction Document Structure Extraction using the GeoLayoutLM model. These documnents were particularly large documents and with no specified key vaue pairs. this worked similar to how the Azure Entity extraction works.

• ChemSpec Spec Sheet Extractions SER Key agnostic NER was performed on chemical spec-sheets using SER model, specifics of the model architecture and configurations can not be shared due to proprietery nature of the software currently in production.
• AutoAnnot Auto Annotations Using LLM's I spearheaded the development of a multi-agent framework for various tasks, our test task was generating automatic annotations from documents, with minimal human supervision we are able to generate a single doucment annotation with an average time of 1.43 ~ 2 seconds with further human correction leading up to a total 7.8 ~ 8 seconds. previously same document was annotated by annotators, personal recorded time for annotating the same document is 94 seconds. This annotation tool offers over 11x speed up for annotators.
• DataSage Document Retrieval Paired with LLM A novel method was introduced for increasing the reference vector data points for document retrieval leading to accurate data retrieval from smaller text chunks paired with LLM's for QA on source Databases. This RAG based system allows our clients to rapidly add new data to over 27k documents database and ask questions from LLM using this database with an average sub-second response.
• Teacher Student Model Trainings LLM used for generating task specific data Larger LLM's acting as the teacher models were used to generate task specific data for various smaller models including LayoutLMV2 to be trained for specific NER and Classification Tasks. Various text processing techniques were used to generate textual and image data.

• LLM Finetune using LoRA LLM Finetune for task specific extraction LLM was finetuned for specific task extractions using LoRA (Low Rank Adaptation). This allowed us to train LLM using relatively low resources and achieve a a convergence on a specific task using LLM's.
• LLM Based Extraction LLM based extraction services using CPU only environment LLM based extraction services built upon llama.cpp, grammar and non grammar based ghost elimination services were introduced for removing any helluciniations from the LLM's. a 100 percent conformity with the source document was established. read more on GBNF (GGML BNF) format here.

• Block Segmentation and Tiling Block segmentation using ocr based closing block segmentation required for training various transformers based algorithms for feeding consistent data blocks leading up to drastic increase in recall in NER tasks. Document tiling and feeding tiling information during training increased the accuracy in various NER Tasks.
• GNN Document Classifier Document Classification GNN Based Document Classifier microservice for lightweight inference in CPU only environemnt. the evaluation on customer data yielded 98.4% accuracy.

• DeepReader AI Data Extraction Automation from German Language Documents NER on German Documents using a Custom Derivative Model from LayoutLMV2, IP owned by the company, architecture can not be open sourced. blog.
• PhelixAI Data Extraction Automation in Medical Insurance Documents Developed a Flask Based WEB API for Medical Document Classification, Medical Object Detection and Medical NER for Automatic Data Filling, We also developed an in house standard for Data Storage, Training Procedures from Database stored Ground Truths and Inference Pipeline. The Medical Document NER was a custom model architecture based on LayoutLM base model architecture and outperformed all the existing models.
• TabAUG WebAPP(Active) Building a Full Scale WebAPP for TabAUG for demonstration The Project contains developing a full scale webapp for processing tables to generate augmented tables. The webapp would also provide the users with their existing tables metadata and augmented tables metadata. Update: Statistics Generation features are all designed and implemented, Augmentation To follow.
• Pneumonia Detection [website] tested and implemented a new form of pre-processing white exposure limits to detect pneumonia, achieved a 92.76% testing accuracy, created a webapp for a client test, currently live on the link given above. Note: The advises generated are static based on various symptom inputs and confidence score of the algorithm.

• FishTally Fish Detection and Counting in Underwater Environment Extending upon Ahsan Jalal's work on fish recognition, my work includes working on combining GMM (gaussian mixture model) background substracted image features with optical flow image features to ehance the original image to be fed into yolov3 darknet and Faster RCNN for fish recognition in video sequences. Another planned addition is utilizing Tracktor++ to track and count species of fishes in a video sequence.
• SynthGen Synthetic Tabular Data Generation For Controlling Model Behavior using UNLV dataset format I am building a tool which allows the user to read their data pool to analyze different charactaristics of their data and then generate more data of similar charactaristics or change them as per their requirement, we're currently testing controlling model behavior with generated data by conducting experiments on Deep Splitting and Merging model.
• Tabsegv2 Table Structure Decomposition with SPLIT model I worked on developing an end to end pipeline for extracting data from scanned table images, We used the earlier developed TableDet detection module using Faster RCNN to detect tables and then used Split Model along with merge heuristics to detect row/column segmenations, once segmented cells were produced and then exported as XML and CSV formatted data.
• RecAARZ Recommendation Engine for AARZ built an API atop Recommendation Engine for a real estate buy and sell platform known by the name of AARZ. We used the matrix factorisation model known as LightFM for building the Recommendation Engine as it outperformed alternative methods in both cold start and sparse interaction matrix scenario.

• TableDet Table Detection in Scanned Document Images [blog] Tables having very little similarity to natural images, hence they were first preprocessed with distance transform in a way to build resemblance to natural images so that we could utilize pretrained ResNet50 backbone, these preprocessed images were then served to a luminoth based Faster RCNN. Starting accruacy achievable with publicaly available models was 52 percent, after feature engineering this figure was increased to 78 percent with >90% IOU and 22 percent partial detection with >80% IOU, with only 3% missed tables with <20% IOU.
• TabSegv1 Table Structure Decomposition with GNN's worked on GNN's (Graph Neural Networks) for decomposing trabular data structure for data extraction autmation in Tabular Data. Proposing a new idea of OCR data closing (OCR bounding box dilation and erosion) for constructing spatial sentences for improving accuracy on CloudSDS Proprietery Dataset.
• Object Concurrency Checking Object Concurrency in Point Cloud Data [Details] Developed a tool for object concurrency in construction process, with real data as a point cloud data and planned data from BIM4D file as an object, we used python OCC library for checking point cloud density inside BIM4d Objects to classify BIM4d objects as existing or non existing in point cloud data. the prototype had a 100% accuracy.
• QuickAnnot Horizontal Vertical Annotations in Tables [Details] Part of Data extraction in tables depend upon where the headings of the tables are situated, defining Horizontal and Vertical tables becomes increasingly important, I developed an annotation tool in Tkinter for UNLV format with a direct focus on speed.
• TransferLight Transfering Highlights from Source PDF to Target PDF A tool for detecting highlighted codes in source PDF document and transferring highlights to a targer PDF document, I used open source libraries PDFMiner and PyPDF2 to recursively traverse PDF Objects to find highlights inside the Source pdf documents and then transfer them in similar methodology to Target pdf documents. The tool was developed for Vactor Manafacturing Inc, prior to the tool the company spent 15 minutes per document processing with manual labor, the tool spent an average 21 seconds per document processing getting it closer to maximizing processing potential.