The main requirement was to gather the data of online teaching platforms, what courses do they offer, how many and which authors are there on each e-learning platform and also the metadata for each course like the categories, course details, and co-authors, schedule, etc.
We did this by storing the sitemap of all these sites in the cloud in a customized JSON format and then triggering an SQL stored procedure which compares the already existing data with the new files update the existing courses if there are any changes or add new data.
→ Creating a common architecture that generalizes scrapping all sitemaps and gathers common data from different sites like Udemy, Udacity, Coursera, etc.
→ Upload large content of scrapped data to Azure cloud and dump the data in the warehouse in Azure.
→SQL procedures to check and update only non-existing data in the warehouse.
Cookie | Duration | Description |
---|---|---|
cookielawinfo-checkbox-analytics | 11 months | This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics". |
cookielawinfo-checkbox-functional | 11 months | The cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional". |
cookielawinfo-checkbox-necessary | 11 months | This cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary". |
cookielawinfo-checkbox-others | 11 months | This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other. |
cookielawinfo-checkbox-performance | 11 months | This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance". |
viewed_cookie_policy | 11 months | The cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data. |
We’ve worked with clients of all sizes, from startups to enterprise brands.