Continuous Integration using Jenkins with Python

Continuous Integration (CI) has changed how we create and deliver software, making it quicker, more efficient, and reducing mistakes. Jenkins, a well-known free tool, is crucial for CI because it automates different parts of building software.

In this blog, we will take a gander at how Jenkins functions impeccably with Python, a famous programming language. We will go through how to set up Jenkins for Python projects, how to configure it, and how to run these projects. This shows how utilizing Jenkins with the Python Development process smoother, increases work efficiency, and keeps the quality of software consistent. This post is useful for everybody, from experienced designers to those simply beginning with CI, offering helpful hints on capitalizing on Jenkins and Python in your CI process.

What is Jenkins?

Jenkins is an open-source automation server that enables developers to integrate, test, and deploy code changes seamlessly. It facilitates continuous integration and continuous delivery (CI/CD) practices, ensuring that software applications are built, tested, and released in a timely and efficient manner.

Prerequisites of integrating Jenkins with Python

Before diving into integrating Jenkins with Python, there are a few key things you’ll need.

Basic Understanding of Python Programming: You should be comfortable with writing and understanding Python code. This knowledge is crucial since we’ll be working with Python projects.

Familiarity with Git Version Control System: Knowing how to use Git is important because Jenkins often interacts with repositories for CI processes.

Essential Requirements

In addition to the basic knowledge, there are some essential tools and setups you need:

Jenkins Installation and Configuration: You should have Jenkins installed and properly configured on your machine or a cloud platform. This is the backbone of the CI process we’re discussing.

Python Development Environment Setup: Your machine should be set up for Python development, including having Python installed along with any necessary libraries and development tools.

With these prerequisites and essentials in place, you’ll be well-prepared to start integrating Jenkins with your Python projects.

Also Learn: Python Environment Variables

Core Functionality of Jenkins

Continuous Integration (CI): Jenkins monitors source code repositories like Git or SVN for code changes. Whenever a change is detected, it automatically triggers a build process, which involves compiling, testing, and analyzing the code. This ensures that the codebase remains stable and free of errors.

Continuous Delivery (CD): Once the build process is complete and the code is deemed stable, Jenkins can automatically deploy the updated code to production environments. This eliminates the need for manual deployment tasks, reducing the time it takes to release new features and bug fixes.

Pipelines: Jenkins utilizes pipelines to define and execute automated workflows for building, testing, and deploying code. Pipelines are composed of stages, each representing a specific step in the workflow. This allows for granular control over the automation process.

Plugins: Jenkins boasts a vast ecosystem of plugins that extend its functionality and integrations with various tools and platforms. Plugins provide a wide range of capabilities, such as code analysis, reporting, and integration with different cloud providers.

Benefits of Using Jenkins

Improved Code Quality: Jenkins automates testing, enabling early detection of bugs and ensuring code quality throughout the development cycle.

Faster Release Cycles: Automated builds and deployments significantly reduce the time it takes to release new features and updates.

Reduced Errors: Automation minimizes human error and ensures consistency in the build, test, and deployment process.

Scalability: Jenkins can handle complex workflows and manage multiple builds simultaneously, making it suitable for large-scale projects.

Cost-Effectiveness: Automation frees up developers’ time, allowing them to focus on core development tasks and reducing overall development costs.

Let’s get started by creating a simple fast API application that sends data to a RabbitMQ queue, and the consumer consumes the data from the queue.

Here’s the link to set up RabbitMQ in Ubuntu 22.04

https://vegastack.com/tutorials/how-to-install-rabbitmq-on-ubuntu-22-04/

Let’s set up the repository in the local. The steps to set up are as follows.

1. Clone the repository from the URL mentioned

https://github.com/shubhamsinha1010/jenkin-learning.git

     2. After that create virtualenv in your system by using the following command 

virtualenv venv

source venv bin/activate

I have named the virtualenv as venv, we can use any name as per our choice.

3. after that, let’s install the requirements using the command  

pip install –r requirements.txt

uvicorn main:app

Now we have a fast API application up and running

Also Learn: Jaeger Integration with Spring Cloud

Let’s integrate Jenkins into our fast API application 

Jenkins requires Java to run. Ensure you have Java installed on your system

sudo apt update

sudo apt install default-jre

Add the Jenkins repository key to your system and install Jenkins

wget -q -O – https://pkg.jenkins.io/debian/jenkins.io.key | sudo apt-key add –

sudo sh -c ‘echo deb http://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list’

sudo apt update

sudo apt install Jenkins

Once installed, start and enable the Jenkins service

sudo systemctl start Jenkins

sudo systemctl enable Jenkins

Jenkins usually runs on port 8080 by default. Open a web browser and navigate to http://localhost:8080 or http://your_server_ip:8080 to access the Jenkins dashboard.

Follow the setup wizard in the browser to complete the installation. You’ll need to retrieve the initial admin password, which can be found in the Jenkins server logs:

sudo cat /var/lib/jenkins/secrets/initialAdminPassword

Go to Jenkins Dashboard -> Manage Jenkins -> Manage Plugins -> Available. Look for plugins related to Python, such as “Python Plugin” or “Pipeline Utility Steps”, and install them.

Now, to create a new job, follow the below steps

Step 1: Create a new job

In Jenkins, go to Manage Jenkins > New Item.

Select Pipeline from the Job Type drop-down menu.

Enter a name for your job, such as “MyPipelineJob”.

Click OK to create the job.

Step 2: Configure the Git repository

On the job configuration page, go to the Source Code Management section.

Select Git from the SCM drop-down menu.

Enter the URL of your git repository in the Repository URL field.

If your repository requires authentication, click the Credentials button and select the appropriate credentials from the list.

Click Save to save the configuration.

Step 3: Define the pipeline script

On the job configuration page, go to the Pipeline Scripts section.

Select Pipeline script from SCM from the Pipeline Source drop-down menu.

Enter the name of the pipeline script file in the Pipeline script path field. For example, if your pipeline script is named pipeline.groovy, enter pipeline.groovy.

Click Save to save the configuration.

Step 4: Trigger the pipeline on commits

On the job configuration page, go to the Build Triggers section.

Select Poll SCM from the Build Triggers drop-down menu.

Enter a value for the Poll SCM frequency field. This value determines how often Jenkins will poll your git repository for changes. For example, if you enter *** * * * ***, Jenkins will poll your repository every minute.

Click Save to save the configuration.

Changes to be made in the Jenkins settings

The following changes should be made in the Jenkins settings

Create a new job: This job will be responsible for running the pipeline whenever a code is committed in a particular Git repository.

Configure the Git repository: This step specifies the URL of the Git repository that you want to monitor for changes.

Define the pipeline script: This step specifies the location of the pipeline script file that contains the code for your pipeline.

Trigger the pipeline on commits: This step tells Jenkins to poll your git repository for changes and run the pipeline whenever a new commit is detected.

Additional notes:

You can also use the Build with Parameters option to allow users to pass parameters to the pipeline when it is triggered.

You can use the Build Authorization option to restrict who can trigger the pipeline.

You can use the Notifications option to configure notifications for pipeline events, such as builds starting and completing.

Let’s configure the webhook in GitHub

Configuring the GitHub repository:

Access repository settings: Navigate to the GitHub repository where you want to enable webhooks. Click on the repository name, then select “Settings” from the dropdown menu.

Enable webhooks: In the settings sidebar, locate the “Integrations” section and click on “Webhook”.

Create a new webhook: Click on the “Add webhook” button.

Provide webhook URL: In the “Payload URL” field, enter the URL of your Jenkins job. This URL will be used to deliver notifications about code changes.

Select content type: Choose the content type for the webhook payload. The preferred option is “application/json”, which sends the payload directly as a JSON object in the POST request.

Optional: Configure webhook secret: For added security, you can set a webhook secret. This secret will be used to verify the authenticity of incoming requests from GitHub.

Select events to trigger webhook: Choose the events that should trigger the webhook. Common options include “push” (for code pushes), “pull_request” (for pull requests), and “delete” (for deleting branches or tags).

Click “Add webhook” to save: Once you’ve configured the webhook settings, click on the “Add webhook” button to save the changes.

Now, it is time to configure Slack with Jenkins which sends a Slack notification whenever a Jenkins job starts, succeeds, or fails

Create a Slack App: In your Slack workspace, navigate to the “Apps” page. Click on the “Create New App” button and select “From Scratch”.

Name your Slack App: Enter a name for your Slack app, such as “Jenkins Notifications”.

Choose app type: Select “Incoming Webhooks” as the app type. This will allow your Jenkins job to send notifications to Slack channels.

Add bot user: Click on the “Add bot user” button. This will create a bot user for your Slack app, which will be used to send notifications.

Copy webhook URL: Note down the webhook URL provided by Slack. This URL will be used in the Jenkins job configuration.

Install Slack app: Click on the “Install to Slack” button and follow the prompts to install the Slack app in your workspace.

Configuring Jenkins:

Install Slack Plugin: In Jenkins, go to Manage Jenkins > Plugins. Search for the “Slack Plugin” and install it.

Configure Slack Global Settings: In Jenkins, go to Manage Jenkins > Configure System. Under the “Slack” section, enter the following information:

Slack Webhook URL: Paste the webhook URL you copied from Slack.

Default Channel: Specify the default Slack channel where notifications should be sent. You can override this channel for specific jobs.

Configure Slack Notifications for a Job: Open the job configuration page for the job you want to receive Slack notifications for.

Slack Notifications: Under the “Post-build Actions” section, select “Slack Notification”.

Channel: Choose the Slack channel where notifications should be sent for this job. You can also use a custom channel syntax like #my-custom-channel.

Condition: Select the events for which you want to send Slack notifications. Options include “Always”, “Success”, “Failure”, “Unstable”, and “Not Built”.

Message: Customize the message that will be sent to Slack. You can use variables like ${JOB_NAME}, ${BUILD_STATUS}, and ${BUILD_URL} to insert job-specific information.

Save Jenkins Job Configuration: Click on the “Save” button to apply the Slack notification settings for the job.

Now, whenever your Jenkins job starts, succeeds, or fails, a Slack notification will be sent to the specified channel, providing you with real-time updates on your build status.

So now, we have a CI-ready Fastapi web application using Jenkins, slack, and GitHub webhooks.

To sum it up, using Jenkins with Python for Continuous Integration can really change the way you develop software. It not only makes things faster but also helps in reducing mistakes, leading to a more efficient and smooth workflow. Automating various steps, like testing and deploying your software, is key to making sure it’s of good quality.

With Jenkins and Python together, developers can create software that’s both trustworthy and strong. Plus, Jenkins has a lot of extra tools and resources that can make your Continuous Integration process even better. It’s a tool that can grow with your needs in today’s software development world. So, I suggest exploring these tools and resources to make the most out of Jenkins in your Python projects and see how far you can go with Continuous Integration.

Also Learn: How to Integrate Apache Kafka with Spring Boot