Friends who play with AI, where do you get your training sets?

wangerxian

Friends who play with AI, where do you get your training sets? [Copy link]

 

Some image recognition training sets can be obtained from the Internet, but most of the images obtained from the Internet contain watermarks. For example, where can we get such training sets for digital recognition in this [AI Challenge]?

Images can be found online. If it is data from a product sensor, can it only be obtained from the product? If it can only be obtained from the product, it seems that there will be no difference in the data, which may cause the final trained model to be non-robust. What experience do you have? Please share it~

秦天qintian0303

The comprehensiveness and representativeness of training data are very important. Image recognition is OK. There are many ways to obtain pictures. If you can't do it, you can take pictures yourself. The abstract data of sensors is more troublesome.

wangerxian

Qintianqintian0303 posted on 2024-6-20 16:38 The comprehensiveness and representativeness of training data are very important. Image recognition is okay. There are many ways to obtain pictures. If you can't do it, just take them yourself. The sensors are abstract...

The sensor seems to be able to collect data only by itself, but the amount seems to be huge.

hellokitty_bean

Crawler? Or just use Feifei's ImageNet?

wangerxian

Crawler? Or use Feifei's ImageNet directly?

Which is the latter? I am ignorant

851779592

I would also like to ask what is Feifei's ImageNet? How to use it? What are the advantages?

wangerxian

I was wondering, can we use AI to help us collect images or produce sensor data?

freebsder

It depends on what data you want. Many places in Damei will provide it, but I’m not sure about Dongda.

For example, you can go to their official website to download meteorological data for previous years, for example, for economics, there are many, for example, for diseases, you can go to their official medical website. And so on.

There are also various AI competition official websites and forums, some of which provide data.

wangerxian

freebsder posted on 2024-6-21 09:46 It depends on what data you want. Many places in Damei will provide it, but I am not sure about Dongda. For example, you can go to their official website to download the meteorological data of previous years...

The suggestion of an official website and forum for AI competitions is a good one. Indeed, many competitions now have data sets.

hellokitty_bean

wangerxian posted on 2024-6-20 21:42 The latter is? I am ignorant

Fei-Fei Li, a beautiful Chinese-American Computer Vision scientist and a well-known scholar...

Former Vice President of Google

hellokitty_bean

851779592 Published on 2024-6-21 09:11 I would like to ask what is Feifei's ImageNet? How to use it? What are the advantages?

Just check ImageNet and you will know... Very well-known in the CV field

851779592

OK, thank you! I have checked the relevant information, it is really great! Very good! Thank you for your recommendation and explanation!

吾妻思萌

Basically it's free, a lot of datasets

If you want to do it yourself, you have to clean the pictures and label them, which is quite troublesome.

I have used Maix Hub and it is quite good, suitable for embedded devices

wangerxian

Azuma Simeng posted on 2024-6-24 10:51 Basically it’s free. If you want to do a lot of datasets by yourself, you have to clean the pictures and annotate them, which is quite troublesome. I’ve used MaixHub and it’s pretty good, ...

I think that going through this process is the only way to truly train the AI model.

吾妻思萌

wangerxian posted on 2024-6-24 15:21 I think that only by going through this process can we really train the AI model.

I don't quite agree.

Preparing the data set is actually tedious and repetitive work, which is meaningless.

The core of AI is the algorithm, which is how to create the recognition matrix.

Including how to reduce noise, various transformations after noise reduction, taking edges, taking feature points, etc., which are studied by the big guys.

吾妻思萌

wangerxian posted on 2024-6-24 15:21 I think that only by going through this process can we really train the AI model.

As for making training sets and test sets, and annotating and scaling images, these are all done by interns.