TGEG

Industry: Financial and Insurance

TGEG

Overview

This is a Financial/Insurance procedure automation platform where the primary goal is to Crawl 11 different websites related to insurance of different types to automate the Login process with OTP verification (2-Factor Authentication) enabled. The platform provides save report files to Amazon AWS s3 and Bypass Captcha, Recaptcha Verification, and security with Advanced Proxy settings.

The System is implemented custom Logging architecture for Failure cases and reporting in the dashboard via APIs. It is Flask based REST APIs to initiate crawlers which is triggered by Laravel Admin Panel on which Agents can work.

Technology

  • Python3
  • Flask
  • Selenium
  • AWS s3
  • AWS Lambda
  • Luminati
  • Laravel Admin panel
  • AntiCaptcha

Key Technical Challenges:

  • Managing the 2-Factor enabled authentication in some websites, create server architecture that doesn’t fail with step by step API execution.
  • Error reporting Logs creation with parallel execution in crawlers.
  • Managing common architecture between all the crawlers of different websites.
  • Multiple captchas solving systems with auto-switching in cases of continuous failures.

Business + Technical Points:

  • Central management systems for Failure Logs.
  • The automatic reporting system for changing the structure of crawlers.
  • The highly scalable system in terms of usage, system must be able to scale itself when resources are not available due to increased load on the server.
  • Independent crawlers with the option to execute them all in a chain with the admin panel.