Web Scrapping Example in Android kotlin

Output:

1/  Create A New Project: ( Web Scrapping Example )

2/ Add dependency to your build.gralde file ( app module )

implementation("org.jsoup:jsoup:1.15.4")

3/ Add permission to your AndroidManifest.xml file

<uses-permission android:name="android.permission.INTERNET" />

4/  Modify your activity_main.xml

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:orientation="vertical"
    android:id="@+id/main"
    android:layout_marginTop="50dp"
    android:padding="16dp">

    <ScrollView
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        >
        <LinearLayout
            android:layout_width="match_parent"
            android:layout_height="wrap_content"
            android:orientation="vertical"
            android:gravity="center"
            >
            <Button
                android:id="@+id/btnScrape"
                android:layout_width="wrap_content"
                android:layout_height="wrap_content"
                android:text="Scrape Web Data" />

            <TextView
                android:id="@+id/tvResult"
                android:lineHeight="30dp"
                android:layout_width="wrap_content"
                android:layout_height="wrap_content"
                android:text="Web Data Will Be Displayed Here"
                android:paddingTop="20dp" />
            <ProgressBar
                android:id="@+id/progressbar"
                android:layout_marginTop="16dp"
                android:layout_width="wrap_content"
                android:layout_height="wrap_content"
                android:visibility="gone"
                />
        </LinearLayout>
    </ScrollView>

</LinearLayout>

5/  Modify your MainActivity.kt

package com.zissofworks.webscrappingkotlin

import android.os.Bundle
import android.os.Handler
import android.os.Looper
import android.view.View
import android.widget.Button
import android.widget.ProgressBar
import android.widget.TextView
import androidx.activity.enableEdgeToEdge
import androidx.appcompat.app.AppCompatActivity
import androidx.core.view.ViewCompat
import androidx.core.view.WindowInsetsCompat
import org.jsoup.Jsoup
import org.jsoup.nodes.Document
import java.io.IOException
import java.util.concurrent.ExecutorService
import java.util.concurrent.Executors


class MainActivity : AppCompatActivity() {
    private lateinit var tvResult: TextView
    private lateinit var btnScrape: Button
    private lateinit var progressBar: ProgressBar

    private val mainHandler: Handler = Handler(Looper.getMainLooper())
    private val executorService: ExecutorService = Executors.newSingleThreadExecutor()

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        enableEdgeToEdge()
        setContentView(R.layout.activity_main)
        ViewCompat.setOnApplyWindowInsetsListener(findViewById(R.id.main)) { v, insets ->
            val systemBars = insets.getInsets(WindowInsetsCompat.Type.systemBars())
            v.setPadding(systemBars.left, systemBars.top, systemBars.right, systemBars.bottom)
            insets
        }

        tvResult = findViewById(R.id.tvResult);
        btnScrape = findViewById(R.id.btnScrape);
        progressBar = findViewById(R.id.progressbar);


        btnScrape.setOnClickListener { startWebScraping() }

    }

    private fun startWebScraping() {
        progressBar.visibility = View.VISIBLE
        executorService.submit {
            val resultText = scrapeWebData()
            mainHandler.post {
                progressBar.visibility = View.GONE
                tvResult.text = resultText
            }
        }
    }

    private fun scrapeWebData(): String {
        var resultText = ""
        try {
            val doc: Document = Jsoup.connect("https://books.toscrape.com").get()


            resultText = doc.text()
        } catch (e: IOException) {
            e.printStackTrace()
            resultText = "Error fetching data!"
        }

        return resultText
    }

    override fun onDestroy() {
        super.onDestroy()
        executorService.shutdown();
        progressBar.setVisibility(View.GONE);
    }
}

1/  For web Scrapping I use the website: https://books.toscrape.com/

    if you want to scrap data from other websites you can replace your site URL here:

//previous url
Document doc = Jsoup.connect("https://books.toscrape.com").get();
//replaced url
Document doc = Jsoup.connect("https://your-url.com").get()