Umar Khan

Machine Learning Engineer and Researcher at DFKI, working on Computer Vision and NLP based systems with Dr. Sheraz Ahmed. My previous works include working with Dr.Faisal Shafait on various Computer Vision R&D and commercial Projects, since my undergrad I have also had a keen focus on Requirements Engineering with my work for aawaz , which proceeded to winning the IBM Pitch your startup cup.

with keen focus on application of research I have worked on numerous commercial projects, you can read more about them here

Interests: Computer Vision, Graph Neural Networks, Autonomous Systems, Biomimicry.

Publications

• WordVIS: A Color Worth A Thousand Words U.Khan, S.Saifullah, S.Agne, A.Dengel, S.Ahmed. [paper] (Currently Used in production, Preprint Only)

• TabAug: Data Driven Augmentation for Enhanced Table Structure Recognition Umar Khan, Sohaib Zahid, Muhammad Asad Ali, Adnan ul Hassan, Faisal Shafait [paper] Publication Status: Accepted at ICDAR2021
• Content-Driven Document Colorization for Table Detection Muhammad Asad Ali, Sohaib Zahid, Umar Khan, Adnan ul Hassan, Faisal Shafait [pre-print] Publication Status: Pre-print

Notable Projects

• ChemSpec Spec Sheet Extractions SER Key agnostic NER was performed on chemical spec-sheets using SER model, specifics of the model architecture and configurations can not be shared due to proprietery nature of the software currently in production.
• AutoAnnot Auto Annotations Using LLM's I spearheaded the development of a multi-agent framework for various tasks, our test task was generating automatic annotations from documents, with minimal human supervision we are able to generate a single doucment annotation with an average time of 1.43 ~ 2 seconds with further humand correction leading up to a total 7.8 ~ 8 seconds. previously same document was annotated by annotators, personal recorded time for annotating the same document is 94 seconds. This annotation tool offers over 11x speed up for annotators.
• DataSage Document Retrieval Paired with LLM A novel method was introduced for increasing the reference vector data points for document retrieval leading to accurate data retrieval from smaller text chunks paired with LLM's for QA on source Databases. This RAG based system allows our clients to rapidly add new data to over 27k documents database and ask questions from LLM using this database with an average sub-second response.
• Teacher Student Model Trainings LLM used for generating task specific data Larger LLM's acting as the teacher models were used to generate task specific data for various smaller models including LayoutLMV2 to be trained for specific NER and Classification Tasks. Various text processing techniques were used to generate textual and image data.

• LLM Finetune using LoRA LLM Finetune for task specific extraction LLM was finetuned for specific task extractions using LoRA (Low Rank Adaptation). This allowed us to train LLM using relatively low resources and achieve a a convergence on a specific task using LLM's.
• LLM Based Extraction LLM based extraction services using CPU only environment LLM based extraction services built upon llama.cpp, grammar and non grammar based ghost elimination services were introduced for removing any helluciniations from the LLM's. a 100 percent conformity with the source document was established. read more on GBNF (GGML BNF) format here.

• Block Segmentation and Tiling Block segmentation using ocr based closing block segmentation required for training various transformers based algorithms for feeding consistent data blocks leading up to drastic increase in recall in NER tasks. Document tiling and feeding tiling information during training increased the accuracy in various NER Tasks.
• GNN Document Classifier Document Classification GNN Based Document Classifier microservice for lightweight inference in CPU only environemnt. the evaluation on customer data yielded 98.4% accuracy.

• DeepReader AI Data Extraction Automation from German Language Documents NER on German Documents using a Custom Derivative Model from LayoutLMV2, IP owned by the company, architecture can not be open sourced. blog.
• PhelixAI Data Extraction Automation in Medical Insurance Documents Developed a Flask Based WEB API for Medical Document Classification, Medical Object Detection and Medical NER for Automatic Data Filling, We also developed an in house standard for Data Storage, Training Procedures from Database stored Ground Truths and Inference Pipeline. The Medical Document NER was a custom model architecture based on LayoutLM base model architecture and outperformed all the existing models.
• TabAUG WebAPP(Active) Building a Full Scale WebAPP for TabAUG for demonstration The Project contains developing a full scale webapp for processing tables to generate augmented tables. The webapp would also provide the users with their existing tables metadata and augmented tables metadata. Update: Statistics Generation features are all designed and implemented, Augmentation To follow.
• Pneumonia Detection [website] tested and implemented a new form of pre-processing white exposure limits to detect pneumonia, achieved a 92.76% testing accuracy, created a webapp for a client test, currently live on the link given above. Note: The advises generated are static based on various symptom inputs and confidence score of the algorithm.

• FishTally Fish Detection and Counting in Underwater Environment Extending upon Ahsan Jalal's work on fish recognition, my work includes working on combining GMM (gaussian mixture model) background substracted image features with optical flow image features to ehance the original image to be fed into yolov3 darknet and Faster RCNN for fish recognition in video sequences. Another planned addition is utilizing Tracktor++ to track and count species of fishes in a video sequence.
• SynthGen Synthetic Tabular Data Generation For Controlling Model Behavior using UNLV dataset format I am building a tool which allows the user to read their data pool to analyze different charactaristics of their data and then generate more data of similar charactaristics or change them as per their requirement, we're currently testing controlling model behavior with generated data by conducting experiments on Deep Splitting and Merging model.
• Tabsegv2 Table Structure Decomposition with SPLIT model I worked on developing an end to end pipeline for extracting data from scanned table images, We used the earlier developed TableDet detection module using Faster RCNN to detect tables and then used Split Model along with merge heuristics to detect row/column segmenations, once segmented cells were produced and then exported as XML and CSV formatted data.
• RecAARZ Recommendation Engine for AARZ built an API atop Recommendation Engine for a real estate buy and sell platform known by the name of AARZ. We used the matrix factorisation model known as LightFM for building the Recommendation Engine as it outperformed alternative methods in both cold start and sparse interaction matrix scenario.

• TableDet Table Detection in Scanned Document Images [blog] Tables having very little similarity to natural images, hence they were first preprocessed with distance transform in a way to build resemblance to natural images so that we could utilize pretrained ResNet50 backbone, these preprocessed images were then served to a luminoth based Faster RCNN. Starting accruacy achievable with publicaly available models was 52 percent, after feature engineering this figure was increased to 78 percent with >90% IOU and 22 percent partial detection with >80% IOU, with only 3% missed tables with <20% IOU.
• TabSegv1 Table Structure Decomposition with GNN's worked on GNN's (Graph Neural Networks) for decomposing trabular data structure for data extraction autmation in Tabular Data. Proposing a new idea of OCR data closing (OCR bounding box dilation and erosion) for constructing spatial sentences for improving accuracy on CloudSDS Proprietery Dataset.
• Object Concurrency Checking Object Concurrency in Point Cloud Data [Details] Developed a tool for object concurrency in construction process, with real data as a point cloud data and planned data from BIM4D file as an object, we used python OCC library for checking point cloud density inside BIM4d Objects to classify BIM4d objects as existing or non existing in point cloud data. the prototype had a 100% accuracy.
• QuickAnnot Horizontal Vertical Annotations in Tables [Details] Part of Data extraction in tables depend upon where the headings of the tables are situated, defining Horizontal and Vertical tables becomes increasingly important, I developed an annotation tool in Tkinter for UNLV format with a direct focus on speed.
• TransferLight Transfering Highlights from Source PDF to Target PDF A tool for detecting highlighted codes in source PDF document and transferring highlights to a targer PDF document, I used open source libraries PDFMiner and PyPDF2 to recursively traverse PDF Objects to find highlights inside the Source pdf documents and then transfer them in similar methodology to Target pdf documents. The tool was developed for Vactor Manafacturing Inc, prior to the tool the company spent 15 minutes per document processing with manual labor, the tool spent an average 21 seconds per document processing getting it closer to maximizing processing potential.