【2024 DigiKey Innovation Contest】

eew_mfMeDT · 2024-11-13 07:33

【2024 DigiKey Innovation Contest】 [Copy link]

Preface

Our system is an intelligent cat behavior monitoring system, which mainly consists of Raspberry Pi and visual sensors. Raspberry Pi is the core processing unit of the system, responsible for running programs, processing data and communication. The visual sensor is installed on the Raspberry Pi to capture the activity of the cat. Through the visual recognition algorithm, the image is processed in real time and the cat's activities are identified and tracked. The Gemini multimodal model is used to analyze these images and videos, identify various behaviors of cats, such as eating, playing, sleeping, etc., and record these behavioral data to form a cat's behavior log. Long-term recording can help identify the daily behavior patterns of cats and provide behavioral health analysis. This system supports remote monitoring. Pet owners can remotely view the cat's behavior records through mobile phones or computers to understand their health status and daily activities. Through behavioral data analysis, the system can detect abnormal behaviors of cats at an early stage and take health intervention measures in time. At the same time, the system's data is stored locally or uploaded to the cloud in an encrypted manner to ensure the privacy and security of the data. This system can not only help pet owners better understand and care for their cats, but also provide valuable data for pet health and behavior research.

design

accomplish

This code creates a web page that allows the user to start the camera and send the video stream to a WebSocket server via WebRTC technology. It first obtains the user's camera stream, then connects to the server via WebSocket and creates a PeerConnection to manage the transmission of the video stream. The code also handles the ICE candidate and signaling process to ensure that the video stream can be transmitted to the server correctly.

<!DOCTYPE html>
<html>
<head>
    <title>WebRTC Camera Stream</title>
    <script src="https://webrtc.github.io/adapter/adapter-latest.js"></script>
</head>
<body>
    <video id="localVideo" autoplay muted></video>
    <button id="startButton">Start</button>
    <script type="text/javascript">
        'use strict';

        const localVideo = document.querySelector('video#localVideo');
        const startButton = document.querySelector('button#startButton');
        let localStream, pc, ws;

        startButton.onclick = async () => {
            try {
                // 获取摄像头流
                const stream = await navigator.mediaDevices.getUserMedia({video: true, audio: false});
                localVideo.srcObject = stream;
                localStream = stream;

                // 初始化WebSocket连接
                ws = new WebSocket('ws://localhost:8765');
                ws.onopen = () => {
                    console.log('WebSocket connection established');
                    
                    // 创建PeerConnection
                    pc = new RTCPeerConnection({
                        iceServers: [
                            {urls: 'stun:stun.l.google.com:19302'}
                        ]
                    });

                    // 添加本地流到PeerConnection
                    localStream.getTracks().forEach(track => pc.addTrack(track, localStream));

                    // 处理ICE候选者
                    pc.onicecandidate = event => {
                        if (event.candidate) {
                            ws.send(JSON.stringify({type: 'candidate', candidate: event.candidate}));
                        }
                    };

                    // 创建offer
                    const offer = await pc.createOffer();
                    await pc.setLocalDescription(offer);

                    // 发送offer到信令服务器
                    ws.send(JSON.stringify({type: 'offer', sdp: pc.localDescription}));
                };

                ws.onmessage = async (event) => {
                    const data = JSON.parse(event.data);

                    if (data.type === 'answer') {
                        // 设置远程描述
                        await pc.setRemoteDescription(new RTCSessionDescription(data));
                    } else if (data.type === 'candidate') {
                        // 添加ICE候选者
                        await pc.addIceCandidate(new RTCIceCandidate(data.candidate));
                    }
                };

                ws.onclose = () => console.log('WebSocket connection closed');
                ws.onerror = (error) => console.error('WebSocket error:', error);

            } catch (e) {
                console.error('getUserMedia() error:', e);
            }
        };
    </script>
</body>
</html>

This code implements a WebSocket server for handling WebRTC video streaming. It creates an asynchronous function to handle the client's offer, sets the remote description, creates and sends the answer, and handles ICE candidates. The server is also responsible for receiving the video track, converting the WebRTC video frames using OpenCV and saving them as images. Finally, the server listens for client signaling messages through WebSocket, handles offers and candidates, and keeps running to receive new connections.

import asyncio
import json
import websockets
from aiortc import RTCPeerConnection, RTCSessionDescription, RTCIceCandidate
import cv2
import numpy as np

pcs = set()
room = 'default-room'

async def handle_offer(offer):
    pc = RTCPeerConnection()
    pcs.add(pc)

    @pc.on("icecandidate")
    async def on_icecandidate(candidate):
        if candidate:
            await send_to_client(json.dumps({"type": "candidate", "candidate": candidate.to_json()}))

    @pc.on("track")
    def on_track(track):
        print("Track %s received" % track.kind)

        # 处理视频帧
        async def process_frame():
            while True:
                frame = await track.recv()
                if frame:
                    # 将WebRTC帧转换为OpenCV格式
                    frame = cv2.cvtColor(np.frombuffer(frame, dtype=np.uint8).reshape((frame.height, frame.width, 3)), cv2.COLOR_RGBA2BGR)
                    
                    # 保存图片
                    cv2.imwrite('received_frame.jpg', frame)
                    print("Frame saved")

        asyncio.ensure_future(process_frame())

    await pc.setRemoteDescription(RTCSessionDescription(offer, type="offer"))
    answer = await pc.createAnswer()
    await pc.setLocalDescription(answer)
    return pc.localDescription.sdp

async def send_to_client(message):
    # 这里应该实现向客户端发送消息的逻辑
    pass

async def signal(websocket, path):
    try:
        async for message in websocket:
            data = json.loads(message)
            if data['type'] == 'offer':
                answer = await handle_offer(data['sdp'])
                await websocket.send(json.dumps({"type": "answer", "sdp": answer}))
            elif data['type'] == 'candidate':
                for pc in pcs:
                    await pc.addIceCandidate(RTCIceCandidate.from_json(data['candidate']))
    finally:
        # 处理客户端断开连接
        pass

start_server = websockets.serve(signal, "localhost", 8765)

asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()

This code implements a simple WebRTC signaling server. It handles client connections and messages through WebSocket. The server allows clients to join rooms and forwards offer, answer, and ICE candidate messages between clients in the same room. When each client connects, it is registered in a dictionary containing information about the room it is in. The server listens for messages from the client and forwards them accordingly based on the message type. When a client disconnects, the server removes it from the dictionary.

import asyncio
import json
import websockets

clients = {}

async def signal(websocket, path):
    try:
        # Register the client
        clients[websocket] = None
        print("New client connected")

        async for message in websocket:
            data = json.loads(message)
            action = data.get('action')

            if action == 'join':
                # Store the room ID for this client
                clients[websocket] = data['room']

            elif action == 'offer' or action == 'answer':
                # Forward the offer or answer to the other client in the same room
                for client in clients:
                    if clients[client] == clients[websocket] and client != websocket:
                        await client.send(json.dumps(data))
                        break

            elif action == 'ice-candidate':
                # Forward ICE candidate to the other client in the same room
                for client in clients:
                    if clients[client] == clients[websocket] and client != websocket:
                        await client.send(json.dumps(data))
                        break

    finally:
        # Unregister the client
        if websocket in clients:
            del clients[websocket]
            print("Client disconnected")

start_server = websockets.serve(signal, "localhost", 8765)

asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()

Then generate a web report from the database

import base64

# 生成HTML内容
def generate_html(imgs, describe):
    html_content = """
    
```html
<!DOCTYPE html>
    <html lang="en">
    <head>
        <meta charset="UTF-8">
        <meta name="viewport" content="width=device-width, initial-scale=1.0">
        <title>图片与描述展示</title>
        <style>
            .image-container {
                display: flex;
                flex-wrap: wrap;
                justify-content: space-around;
            }
            .image-item {
                border: 1px solid #ddd;
                border-radius: 8px;
                padding: 10px;
                margin: 10px;
                text-align: center;
            }
            .image-item img {
                max-width: 300px;
                max-height: 300px;
                width: auto;
                height: auto;
            }
        </style>
    </head>
    <body>
        <div class="image-container">
    """
    
    for img, desc in zip(imgs, describe):
        # 将base64编码的图片数据插入到img标签中
        html_content += f"""
            <div class="image-item">
                <img src="data:image/jpeg;base64,{img}" alt="{desc}">
                <p>{desc}</p>
            </div>
        """

    html_content += """
        </div>
    </body>
    </html>

Test video acquisition.

https://www.bilibili.com/video/BV1pA4m1V7WP/?spm_id_from=333.337.search-card.all.click&vd_source=c13e5621e1c5e79fc4965d6a679342eb

We used the you-get project to obtain the test video above.

Jacktang · Published on 2024-11-13 07:33

I would like to know the process of ICE candidate and signaling process code implementation

【2024 DigiKey Innovation Contest】 [Copy link]

Latest reply